Activation Maximization on VGGNet

Dense Layer Visualizations

To visualize activation over final dense layer outputs, we need to switch the softmax activation out for linear since gradient of output node will depend on all the other node activations. Doing this in keras is tricky, so we provide utils.apply_modifications to modify network parameters and rebuild the graph.

If this swapping is not done, the results might be suboptimal. We will start by swapping out 'softmax' for 'linear' and compare what happens if we dont do this at the end.

from keras.applications import VGG16
from vis.utils import utils
from keras import activations

# Build the VGG16 network with ImageNet weights
model = VGG16(weights='imagenet', include_top=True)

# Utility to search for layer index by name. 
# Alternatively we can specify this as -1 since it corresponds to the last layer.
layer_idx = utils.find_layer_idx(model, 'predictions')

# Swap softmax with linear
model.layers[layer_idx].activation = activations.linear
model = utils.apply_modifications(model)

Using TensorFlow backend.
C:\Users\admin\Anaconda3\lib\site-packages\keras\models.py:245: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.
  warnings.warn('No training configuration found in save file: '

Visualizing a specific output category

Lets try visualizing a specific output category. We will pick ouzel which corresponds to imagenet category 20

from vis.visualization import visualize_activation

from matplotlib import pyplot as plt
%matplotlib inline
plt.rcParams['figure.figsize'] = (18, 6)

# 20 is the imagenet category for 'ouzel'
img = visualize_activation(model, layer_idx, filter_indices=208)
plt.imshow(img)

<matplotlib.image.AxesImage at 0x206a89dac88>

Hmm, that sort of looks like a bird. Lets see if we can get better results with more iterations. This time, lets see the verbose output during the optimization process.

# 20 is the imagenet category for 'ouzel'
img = visualize_activation(model, layer_idx, filter_indices=208, max_iter=500, verbose=True)
plt.imshow(img)

Iteration: 1, named_losses: [('ActivationMax Loss', 0.97141623),
 ('L-6.0 Norm Loss', 0.063258268),
 ('TV(2.0) Loss', 6431.5322)], overall loss: 6432.56689453125
Iteration: 2, named_losses: [('ActivationMax Loss', 0.79049647),
 ('L-6.0 Norm Loss', 0.062634453),
 ('TV(2.0) Loss', 3170.3511)], overall loss: 3171.2041015625
Iteration: 3, named_losses: [('ActivationMax Loss', 0.58269399),
 ('L-6.0 Norm Loss', 0.062289629),
 ('TV(2.0) Loss', 1654.5631)], overall loss: 1655.2081298828125
Iteration: 4, named_losses: [('ActivationMax Loss', 0.22189793),
 ('L-6.0 Norm Loss', 0.062079947),
 ('TV(2.0) Loss', 846.2395)], overall loss: 846.5234985351562
Iteration: 5, named_losses: [('ActivationMax Loss', -0.75232071),
 ('L-6.0 Norm Loss', 0.061954714),
 ('TV(2.0) Loss', 414.39957)], overall loss: 413.7091979980469
Iteration: 6, named_losses: [('ActivationMax Loss', -2.228792),
 ('L-6.0 Norm Loss', 0.0618819),
 ('TV(2.0) Loss', 216.3212)], overall loss: 214.15428161621094
Iteration: 7, named_losses: [('ActivationMax Loss', -4.6428308),
 ('L-6.0 Norm Loss', 0.061851799),
 ('TV(2.0) Loss', 155.15286)], overall loss: 150.57188415527344
Iteration: 8, named_losses: [('ActivationMax Loss', -6.0633264),
 ('L-6.0 Norm Loss', 0.061832588),
 ('TV(2.0) Loss', 108.94821)], overall loss: 102.94671630859375
Iteration: 9, named_losses: [('ActivationMax Loss', -7.5877285),
 ('L-6.0 Norm Loss', 0.061822157),
 ('TV(2.0) Loss', 84.623146)], overall loss: 77.09723663330078
Iteration: 10, named_losses: [('ActivationMax Loss', -8.2877169),
 ('L-6.0 Norm Loss', 0.061814107),
 ('TV(2.0) Loss', 68.429367)], overall loss: 60.20346450805664
Iteration: 11, named_losses: [('ActivationMax Loss', -9.5619106),
 ('L-6.0 Norm Loss', 0.061809376),
 ('TV(2.0) Loss', 49.435028)], overall loss: 39.93492889404297
Iteration: 12, named_losses: [('ActivationMax Loss', -11.232865),
 ('L-6.0 Norm Loss', 0.061808772),
 ('TV(2.0) Loss', 58.409405)], overall loss: 47.23834991455078
Iteration: 13, named_losses: [('ActivationMax Loss', -12.306953),
 ('L-6.0 Norm Loss', 0.061803903),
 ('TV(2.0) Loss', 43.303444)], overall loss: 31.05829429626465
Iteration: 14, named_losses: [('ActivationMax Loss', -13.237044),
 ('L-6.0 Norm Loss', 0.061803937),
 ('TV(2.0) Loss', 44.820957)], overall loss: 31.64571762084961
Iteration: 15, named_losses: [('ActivationMax Loss', -14.303227),
 ('L-6.0 Norm Loss', 0.061802477),
 ('TV(2.0) Loss', 38.886562)], overall loss: 24.645137786865234
Iteration: 16, named_losses: [('ActivationMax Loss', -13.728777),
 ('L-6.0 Norm Loss', 0.061802521),
 ('TV(2.0) Loss', 34.364601)], overall loss: 20.697628021240234
Iteration: 17, named_losses: [('ActivationMax Loss', -14.787039),
 ('L-6.0 Norm Loss', 0.061803594),
 ('TV(2.0) Loss', 33.592503)], overall loss: 18.867267608642578
Iteration: 18, named_losses: [('ActivationMax Loss', -15.779641),
 ('L-6.0 Norm Loss', 0.06180406),
 ('TV(2.0) Loss', 34.08147)], overall loss: 18.363632202148438
Iteration: 19, named_losses: [('ActivationMax Loss', -16.478008),
 ('L-6.0 Norm Loss', 0.061804332),
 ('TV(2.0) Loss', 32.013016)], overall loss: 15.596811294555664
Iteration: 20, named_losses: [('ActivationMax Loss', -15.055806),
 ('L-6.0 Norm Loss', 0.061805747),
 ('TV(2.0) Loss', 29.684216)], overall loss: 14.690215110778809
Iteration: 21, named_losses: [('ActivationMax Loss', -16.137663),
 ('L-6.0 Norm Loss', 0.061806496),
 ('TV(2.0) Loss', 28.671921)], overall loss: 12.596063613891602
Iteration: 22, named_losses: [('ActivationMax Loss', -14.804443),
 ('L-6.0 Norm Loss', 0.061806872),
 ('TV(2.0) Loss', 27.789173)], overall loss: 13.046536445617676
Iteration: 23, named_losses: [('ActivationMax Loss', -19.267853),
 ('L-6.0 Norm Loss', 0.061808273),
 ('TV(2.0) Loss', 27.415991)], overall loss: 8.209945678710938
Iteration: 24, named_losses: [('ActivationMax Loss', -18.574982),
 ('L-6.0 Norm Loss', 0.061808854),
 ('TV(2.0) Loss', 32.581467)], overall loss: 14.068294525146484
Iteration: 25, named_losses: [('ActivationMax Loss', -19.745275),
 ('L-6.0 Norm Loss', 0.061810944),
 ('TV(2.0) Loss', 21.485073)], overall loss: 1.8016090393066406
Iteration: 26, named_losses: [('ActivationMax Loss', -18.792763),
 ('L-6.0 Norm Loss', 0.061813042),
 ('TV(2.0) Loss', 28.68701)], overall loss: 9.956060409545898
Iteration: 27, named_losses: [('ActivationMax Loss', -22.00914),
 ('L-6.0 Norm Loss', 0.061815001),
 ('TV(2.0) Loss', 23.565722)], overall loss: 1.6183967590332031
Iteration: 28, named_losses: [('ActivationMax Loss', -21.074915),
 ('L-6.0 Norm Loss', 0.061816759),
 ('TV(2.0) Loss', 28.65246)], overall loss: 7.639362335205078
Iteration: 29, named_losses: [('ActivationMax Loss', -22.486017),
 ('L-6.0 Norm Loss', 0.061817624),
 ('TV(2.0) Loss', 22.942413)], overall loss: 0.5182132720947266
Iteration: 30, named_losses: [('ActivationMax Loss', -21.193037),
 ('L-6.0 Norm Loss', 0.061819516),
 ('TV(2.0) Loss', 23.438011)], overall loss: 2.306793212890625
Iteration: 31, named_losses: [('ActivationMax Loss', -23.776245),
 ('L-6.0 Norm Loss', 0.061821155),
 ('TV(2.0) Loss', 26.53076)], overall loss: 2.816335678100586
Iteration: 32, named_losses: [('ActivationMax Loss', -22.899809),
 ('L-6.0 Norm Loss', 0.061821859),
 ('TV(2.0) Loss', 26.275873)], overall loss: 3.437885284423828
Iteration: 33, named_losses: [('ActivationMax Loss', -26.173943),
 ('L-6.0 Norm Loss', 0.061823554),
 ('TV(2.0) Loss', 25.72094)], overall loss: -0.39118003845214844
Iteration: 34, named_losses: [('ActivationMax Loss', -26.932833),
 ('L-6.0 Norm Loss', 0.0618251),
 ('TV(2.0) Loss', 26.088915)], overall loss: -0.7820930480957031
Iteration: 35, named_losses: [('ActivationMax Loss', -27.031651),
 ('L-6.0 Norm Loss', 0.061826348),
 ('TV(2.0) Loss', 25.547386)], overall loss: -1.4224376678466797
Iteration: 36, named_losses: [('ActivationMax Loss', -24.701706),
 ('L-6.0 Norm Loss', 0.061828248),
 ('TV(2.0) Loss', 24.852983)], overall loss: 0.2131061553955078
Iteration: 37, named_losses: [('ActivationMax Loss', -29.244251),
 ('L-6.0 Norm Loss', 0.061830334),
 ('TV(2.0) Loss', 24.616421)], overall loss: -4.565999984741211
Iteration: 38, named_losses: [('ActivationMax Loss', -29.397734),
 ('L-6.0 Norm Loss', 0.061831027),
 ('TV(2.0) Loss', 27.935814)], overall loss: -1.4000892639160156
Iteration: 39, named_losses: [('ActivationMax Loss', -31.532206),
 ('L-6.0 Norm Loss', 0.061833579),
 ('TV(2.0) Loss', 24.517387)], overall loss: -6.952983856201172
Iteration: 40, named_losses: [('ActivationMax Loss', -26.926643),
 ('L-6.0 Norm Loss', 0.061833564),
 ('TV(2.0) Loss', 27.276613)], overall loss: 0.41180419921875
Iteration: 41, named_losses: [('ActivationMax Loss', -32.598354),
 ('L-6.0 Norm Loss', 0.061836634),
 ('TV(2.0) Loss', 23.962643)], overall loss: -8.573875427246094
Iteration: 42, named_losses: [('ActivationMax Loss', -32.330879),
 ('L-6.0 Norm Loss', 0.061836053),
 ('TV(2.0) Loss', 29.166531)], overall loss: -3.1025123596191406
Iteration: 43, named_losses: [('ActivationMax Loss', -33.610107),
 ('L-6.0 Norm Loss', 0.06183926),
 ('TV(2.0) Loss', 23.404312)], overall loss: -10.14395523071289
Iteration: 44, named_losses: [('ActivationMax Loss', -33.274673),
 ('L-6.0 Norm Loss', 0.061840199),
 ('TV(2.0) Loss', 27.138779)], overall loss: -6.074054718017578
Iteration: 45, named_losses: [('ActivationMax Loss', -34.783062),
 ('L-6.0 Norm Loss', 0.061843205),
 ('TV(2.0) Loss', 24.915802)], overall loss: -9.805416107177734
Iteration: 46, named_losses: [('ActivationMax Loss', -34.768463),
 ('L-6.0 Norm Loss', 0.06184417),
 ('TV(2.0) Loss', 28.046333)], overall loss: -6.660285949707031
Iteration: 47, named_losses: [('ActivationMax Loss', -34.694176),
 ('L-6.0 Norm Loss', 0.061846826),
 ('TV(2.0) Loss', 24.260517)], overall loss: -10.371810913085938
Iteration: 48, named_losses: [('ActivationMax Loss', -34.60387),
 ('L-6.0 Norm Loss', 0.061848208),
 ('TV(2.0) Loss', 26.81575)], overall loss: -7.7262725830078125
Iteration: 49, named_losses: [('ActivationMax Loss', -34.355839),
 ('L-6.0 Norm Loss', 0.061851349),
 ('TV(2.0) Loss', 25.69698)], overall loss: -8.597007751464844
Iteration: 50, named_losses: [('ActivationMax Loss', -37.314697),
 ('L-6.0 Norm Loss', 0.061852284),
 ('TV(2.0) Loss', 27.328043)], overall loss: -9.924802780151367
Iteration: 51, named_losses: [('ActivationMax Loss', -34.876122),
 ('L-6.0 Norm Loss', 0.061853487),
 ('TV(2.0) Loss', 27.724331)], overall loss: -7.089935302734375
Iteration: 52, named_losses: [('ActivationMax Loss', -39.449348),
 ('L-6.0 Norm Loss', 0.061854906),
 ('TV(2.0) Loss', 25.749723)], overall loss: -13.63776969909668
Iteration: 53, named_losses: [('ActivationMax Loss', -34.507114),
 ('L-6.0 Norm Loss', 0.061856672),
 ('TV(2.0) Loss', 28.472267)], overall loss: -5.972991943359375
Iteration: 54, named_losses: [('ActivationMax Loss', -38.422012),
 ('L-6.0 Norm Loss', 0.061859582),
 ('TV(2.0) Loss', 25.59861)], overall loss: -12.761543273925781
Iteration: 55, named_losses: [('ActivationMax Loss', -37.630882),
 ('L-6.0 Norm Loss', 0.061862081),
 ('TV(2.0) Loss', 25.904926)], overall loss: -11.664093017578125
Iteration: 56, named_losses: [('ActivationMax Loss', -39.460823),
 ('L-6.0 Norm Loss', 0.061863963),
 ('TV(2.0) Loss', 25.940382)], overall loss: -13.458578109741211
Iteration: 57, named_losses: [('ActivationMax Loss', -37.44783),
 ('L-6.0 Norm Loss', 0.061864723),
 ('TV(2.0) Loss', 25.841259)], overall loss: -11.544708251953125
Iteration: 58, named_losses: [('ActivationMax Loss', -38.112488),
 ('L-6.0 Norm Loss', 0.061866),
 ('TV(2.0) Loss', 26.54484)], overall loss: -11.505781173706055
Iteration: 59, named_losses: [('ActivationMax Loss', -39.68034),
 ('L-6.0 Norm Loss', 0.061868262),
 ('TV(2.0) Loss', 26.192095)], overall loss: -13.42637825012207
Iteration: 60, named_losses: [('ActivationMax Loss', -40.557468),
 ('L-6.0 Norm Loss', 0.061870426),
 ('TV(2.0) Loss', 26.508945)], overall loss: -13.986652374267578
Iteration: 61, named_losses: [('ActivationMax Loss', -37.502995),
 ('L-6.0 Norm Loss', 0.06187167),
 ('TV(2.0) Loss', 27.563484)], overall loss: -9.877639770507812
Iteration: 62, named_losses: [('ActivationMax Loss', -42.764538),
 ('L-6.0 Norm Loss', 0.061873868),
 ('TV(2.0) Loss', 28.02549)], overall loss: -14.677173614501953
Iteration: 63, named_losses: [('ActivationMax Loss', -40.308819),
 ('L-6.0 Norm Loss', 0.061874345),
 ('TV(2.0) Loss', 27.998236)], overall loss: -12.248708724975586
Iteration: 64, named_losses: [('ActivationMax Loss', -43.451042),
 ('L-6.0 Norm Loss', 0.061877608),
 ('TV(2.0) Loss', 27.283531)], overall loss: -16.105632781982422
Iteration: 65, named_losses: [('ActivationMax Loss', -41.049618),
 ('L-6.0 Norm Loss', 0.061878487),
 ('TV(2.0) Loss', 28.622091)], overall loss: -12.36564826965332
Iteration: 66, named_losses: [('ActivationMax Loss', -44.373859),
 ('L-6.0 Norm Loss', 0.061881226),
 ('TV(2.0) Loss', 28.397924)], overall loss: -15.914052963256836
Iteration: 67, named_losses: [('ActivationMax Loss', -41.850113),
 ('L-6.0 Norm Loss', 0.061883464),
 ('TV(2.0) Loss', 28.680431)], overall loss: -13.107799530029297
Iteration: 68, named_losses: [('ActivationMax Loss', -45.342937),
 ('L-6.0 Norm Loss', 0.061886776),
 ('TV(2.0) Loss', 28.090237)], overall loss: -17.190814971923828
Iteration: 69, named_losses: [('ActivationMax Loss', -44.038921),
 ('L-6.0 Norm Loss', 0.061888404),
 ('TV(2.0) Loss', 29.612701)], overall loss: -14.364330291748047
Iteration: 70, named_losses: [('ActivationMax Loss', -42.084229),
 ('L-6.0 Norm Loss', 0.061890252),
 ('TV(2.0) Loss', 28.791359)], overall loss: -13.230979919433594
Iteration: 71, named_losses: [('ActivationMax Loss', -44.365646),
 ('L-6.0 Norm Loss', 0.061892226),
 ('TV(2.0) Loss', 28.481213)], overall loss: -15.822540283203125
Iteration: 72, named_losses: [('ActivationMax Loss', -45.506336),
 ('L-6.0 Norm Loss', 0.061894782),
 ('TV(2.0) Loss', 28.660917)], overall loss: -16.783525466918945
Iteration: 73, named_losses: [('ActivationMax Loss', -43.627838),
 ('L-6.0 Norm Loss', 0.061896794),
 ('TV(2.0) Loss', 28.875906)], overall loss: -14.690034866333008
Iteration: 74, named_losses: [('ActivationMax Loss', -48.034088),
 ('L-6.0 Norm Loss', 0.061899066),
 ('TV(2.0) Loss', 29.591539)], overall loss: -18.380651473999023
Iteration: 75, named_losses: [('ActivationMax Loss', -47.784859),
 ('L-6.0 Norm Loss', 0.061901327),
 ('TV(2.0) Loss', 30.418293)], overall loss: -17.304664611816406
Iteration: 76, named_losses: [('ActivationMax Loss', -48.509724),
 ('L-6.0 Norm Loss', 0.061903108),
 ('TV(2.0) Loss', 31.360958)], overall loss: -17.08686065673828
Iteration: 77, named_losses: [('ActivationMax Loss', -51.146233),
 ('L-6.0 Norm Loss', 0.061906166),
 ('TV(2.0) Loss', 30.863258)], overall loss: -20.2210693359375
Iteration: 78, named_losses: [('ActivationMax Loss', -49.92968),
 ('L-6.0 Norm Loss', 0.061908245),
 ('TV(2.0) Loss', 33.529434)], overall loss: -16.338336944580078
Iteration: 79, named_losses: [('ActivationMax Loss', -50.430534),
 ('L-6.0 Norm Loss', 0.061910041),
 ('TV(2.0) Loss', 30.164005)], overall loss: -20.204620361328125
Iteration: 80, named_losses: [('ActivationMax Loss', -52.95528),
 ('L-6.0 Norm Loss', 0.061913066),
 ('TV(2.0) Loss', 32.00169)], overall loss: -20.891677856445312
Iteration: 81, named_losses: [('ActivationMax Loss', -49.243568),
 ('L-6.0 Norm Loss', 0.061915413),
 ('TV(2.0) Loss', 31.256165)], overall loss: -17.925487518310547
Iteration: 82, named_losses: [('ActivationMax Loss', -50.675076),
 ('L-6.0 Norm Loss', 0.061916973),
 ('TV(2.0) Loss', 30.083063)], overall loss: -20.53009605407715
Iteration: 83, named_losses: [('ActivationMax Loss', -45.578045),
 ('L-6.0 Norm Loss', 0.061919697),
 ('TV(2.0) Loss', 29.900642)], overall loss: -15.615482330322266
Iteration: 84, named_losses: [('ActivationMax Loss', -51.546715),
 ('L-6.0 Norm Loss', 0.061922215),
 ('TV(2.0) Loss', 29.485228)], overall loss: -21.999563217163086
Iteration: 85, named_losses: [('ActivationMax Loss', -49.768791),
 ('L-6.0 Norm Loss', 0.061922707),
 ('TV(2.0) Loss', 30.000303)], overall loss: -19.70656394958496
Iteration: 86, named_losses: [('ActivationMax Loss', -51.084656),
 ('L-6.0 Norm Loss', 0.061927199),
 ('TV(2.0) Loss', 29.610769)], overall loss: -21.411958694458008
Iteration: 87, named_losses: [('ActivationMax Loss', -51.727654),
 ('L-6.0 Norm Loss', 0.061928786),
 ('TV(2.0) Loss', 29.891582)], overall loss: -21.77414321899414
Iteration: 88, named_losses: [('ActivationMax Loss', -53.253651),
 ('L-6.0 Norm Loss', 0.061932579),
 ('TV(2.0) Loss', 29.866047)], overall loss: -23.325672149658203
Iteration: 89, named_losses: [('ActivationMax Loss', -53.444038),
 ('L-6.0 Norm Loss', 0.061934002),
 ('TV(2.0) Loss', 31.395111)], overall loss: -21.98699188232422
Iteration: 90, named_losses: [('ActivationMax Loss', -52.004948),
 ('L-6.0 Norm Loss', 0.061937891),
 ('TV(2.0) Loss', 31.637974)], overall loss: -20.305034637451172
Iteration: 91, named_losses: [('ActivationMax Loss', -54.083946),
 ('L-6.0 Norm Loss', 0.061939374),
 ('TV(2.0) Loss', 31.597507)], overall loss: -22.42449951171875
Iteration: 92, named_losses: [('ActivationMax Loss', -53.285637),
 ('L-6.0 Norm Loss', 0.061943945),
 ('TV(2.0) Loss', 31.925699)], overall loss: -21.29799461364746
Iteration: 93, named_losses: [('ActivationMax Loss', -56.73558),
 ('L-6.0 Norm Loss', 0.06194441),
 ('TV(2.0) Loss', 32.712444)], overall loss: -23.961193084716797
Iteration: 94, named_losses: [('ActivationMax Loss', -55.262321),
 ('L-6.0 Norm Loss', 0.061947159),
 ('TV(2.0) Loss', 34.180748)], overall loss: -21.01962661743164
Iteration: 95, named_losses: [('ActivationMax Loss', -58.03009),
 ('L-6.0 Norm Loss', 0.061948061),
 ('TV(2.0) Loss', 34.587673)], overall loss: -23.380470275878906
Iteration: 96, named_losses: [('ActivationMax Loss', -58.430077),
 ('L-6.0 Norm Loss', 0.061951797),
 ('TV(2.0) Loss', 34.242279)], overall loss: -24.12584686279297
Iteration: 97, named_losses: [('ActivationMax Loss', -58.134132),
 ('L-6.0 Norm Loss', 0.061953135),
 ('TV(2.0) Loss', 35.653656)], overall loss: -22.418521881103516
Iteration: 98, named_losses: [('ActivationMax Loss', -59.621025),
 ('L-6.0 Norm Loss', 0.061956406),
 ('TV(2.0) Loss', 34.989204)], overall loss: -24.569862365722656
Iteration: 99, named_losses: [('ActivationMax Loss', -58.107517),
 ('L-6.0 Norm Loss', 0.061958894),
 ('TV(2.0) Loss', 34.042633)], overall loss: -24.002925872802734
Iteration: 100, named_losses: [('ActivationMax Loss', -60.73082),
 ('L-6.0 Norm Loss', 0.061960611),
 ('TV(2.0) Loss', 35.640072)], overall loss: -25.028785705566406
Iteration: 101, named_losses: [('ActivationMax Loss', -57.405891),
 ('L-6.0 Norm Loss', 0.061962139),
 ('TV(2.0) Loss', 34.265808)], overall loss: -23.078121185302734
Iteration: 102, named_losses: [('ActivationMax Loss', -56.496853),
 ('L-6.0 Norm Loss', 0.061965242),
 ('TV(2.0) Loss', 33.679848)], overall loss: -22.75503921508789
Iteration: 103, named_losses: [('ActivationMax Loss', -58.354759),
 ('L-6.0 Norm Loss', 0.061968133),
 ('TV(2.0) Loss', 33.263851)], overall loss: -25.02893829345703
Iteration: 104, named_losses: [('ActivationMax Loss', -55.909645),
 ('L-6.0 Norm Loss', 0.061969269),
 ('TV(2.0) Loss', 33.181484)], overall loss: -22.66619110107422
Iteration: 105, named_losses: [('ActivationMax Loss', -59.88998),
 ('L-6.0 Norm Loss', 0.061972596),
 ('TV(2.0) Loss', 33.153492)], overall loss: -26.674514770507812
Iteration: 106, named_losses: [('ActivationMax Loss', -57.142193),
 ('L-6.0 Norm Loss', 0.061975535),
 ('TV(2.0) Loss', 33.053875)], overall loss: -24.02634048461914
Iteration: 107, named_losses: [('ActivationMax Loss', -61.451839),
 ('L-6.0 Norm Loss', 0.061977036),
 ('TV(2.0) Loss', 33.805119)], overall loss: -27.58474349975586
Iteration: 108, named_losses: [('ActivationMax Loss', -59.287399),
 ('L-6.0 Norm Loss', 0.061980143),
 ('TV(2.0) Loss', 34.294693)], overall loss: -24.93072509765625
Iteration: 109, named_losses: [('ActivationMax Loss', -60.47403),
 ('L-6.0 Norm Loss', 0.061982408),
 ('TV(2.0) Loss', 34.443748)], overall loss: -25.968299865722656
Iteration: 110, named_losses: [('ActivationMax Loss', -58.71207),
 ('L-6.0 Norm Loss', 0.061985631),
 ('TV(2.0) Loss', 33.906826)], overall loss: -24.74325942993164
Iteration: 111, named_losses: [('ActivationMax Loss', -61.806866),
 ('L-6.0 Norm Loss', 0.061985873),
 ('TV(2.0) Loss', 34.244976)], overall loss: -27.49990463256836
Iteration: 112, named_losses: [('ActivationMax Loss', -62.836121),
 ('L-6.0 Norm Loss', 0.061989412),
 ('TV(2.0) Loss', 34.783371)], overall loss: -27.990760803222656
Iteration: 113, named_losses: [('ActivationMax Loss', -62.256119),
 ('L-6.0 Norm Loss', 0.06199199),
 ('TV(2.0) Loss', 35.825348)], overall loss: -26.368778228759766
Iteration: 114, named_losses: [('ActivationMax Loss', -63.103291),
 ('L-6.0 Norm Loss', 0.061995275),
 ('TV(2.0) Loss', 36.541943)], overall loss: -26.499351501464844
Iteration: 115, named_losses: [('ActivationMax Loss', -66.17952),
 ('L-6.0 Norm Loss', 0.061996721),
 ('TV(2.0) Loss', 36.475121)], overall loss: -29.64240264892578
Iteration: 116, named_losses: [('ActivationMax Loss', -63.065716),
 ('L-6.0 Norm Loss', 0.061999388),
 ('TV(2.0) Loss', 37.569038)], overall loss: -25.434677124023438
Iteration: 117, named_losses: [('ActivationMax Loss', -66.343987),
 ('L-6.0 Norm Loss', 0.06200226),
 ('TV(2.0) Loss', 37.390579)], overall loss: -28.891403198242188
Iteration: 118, named_losses: [('ActivationMax Loss', -63.874794),
 ('L-6.0 Norm Loss', 0.062005922),
 ('TV(2.0) Loss', 38.222549)], overall loss: -25.590240478515625
Iteration: 119, named_losses: [('ActivationMax Loss', -66.944595),
 ('L-6.0 Norm Loss', 0.062006354),
 ('TV(2.0) Loss', 37.998325)], overall loss: -28.884265899658203
Iteration: 120, named_losses: [('ActivationMax Loss', -66.144707),
 ('L-6.0 Norm Loss', 0.062009647),
 ('TV(2.0) Loss', 37.894978)], overall loss: -28.18771743774414
Iteration: 121, named_losses: [('ActivationMax Loss', -67.952347),
 ('L-6.0 Norm Loss', 0.062010735),
 ('TV(2.0) Loss', 37.942554)], overall loss: -29.94778060913086
Iteration: 122, named_losses: [('ActivationMax Loss', -64.372932),
 ('L-6.0 Norm Loss', 0.062014997),
 ('TV(2.0) Loss', 37.322201)], overall loss: -26.988719940185547
Iteration: 123, named_losses: [('ActivationMax Loss', -68.044655),
 ('L-6.0 Norm Loss', 0.062015798),
 ('TV(2.0) Loss', 36.492744)], overall loss: -31.489891052246094
Iteration: 124, named_losses: [('ActivationMax Loss', -65.490463),
 ('L-6.0 Norm Loss', 0.062019829),
 ('TV(2.0) Loss', 38.379013)], overall loss: -27.04943084716797
Iteration: 125, named_losses: [('ActivationMax Loss', -68.026848),
 ('L-6.0 Norm Loss', 0.062022008),
 ('TV(2.0) Loss', 36.75156)], overall loss: -31.213268280029297
Iteration: 126, named_losses: [('ActivationMax Loss', -66.589096),
 ('L-6.0 Norm Loss', 0.06202409),
 ('TV(2.0) Loss', 36.916958)], overall loss: -29.610111236572266
Iteration: 127, named_losses: [('ActivationMax Loss', -68.644791),
 ('L-6.0 Norm Loss', 0.062028624),
 ('TV(2.0) Loss', 36.316494)], overall loss: -32.26626968383789
Iteration: 128, named_losses: [('ActivationMax Loss', -67.25325),
 ('L-6.0 Norm Loss', 0.062029175),
 ('TV(2.0) Loss', 36.969902)], overall loss: -30.22132110595703
Iteration: 129, named_losses: [('ActivationMax Loss', -68.111145),
 ('L-6.0 Norm Loss', 0.062034544),
 ('TV(2.0) Loss', 37.656475)], overall loss: -30.392635345458984
Iteration: 130, named_losses: [('ActivationMax Loss', -67.144363),
 ('L-6.0 Norm Loss', 0.06203473),
 ('TV(2.0) Loss', 37.263889)], overall loss: -29.818439483642578
Iteration: 131, named_losses: [('ActivationMax Loss', -69.87751),
 ('L-6.0 Norm Loss', 0.062040146),
 ('TV(2.0) Loss', 37.438309)], overall loss: -32.377159118652344
Iteration: 132, named_losses: [('ActivationMax Loss', -68.698944),
 ('L-6.0 Norm Loss', 0.062041208),
 ('TV(2.0) Loss', 38.115288)], overall loss: -30.52161407470703
Iteration: 133, named_losses: [('ActivationMax Loss', -69.748329),
 ('L-6.0 Norm Loss', 0.062045105),
 ('TV(2.0) Loss', 38.165726)], overall loss: -31.52056121826172
Iteration: 134, named_losses: [('ActivationMax Loss', -69.662468),
 ('L-6.0 Norm Loss', 0.062046837),
 ('TV(2.0) Loss', 38.984352)], overall loss: -30.616065979003906
Iteration: 135, named_losses: [('ActivationMax Loss', -68.384277),
 ('L-6.0 Norm Loss', 0.062050477),
 ('TV(2.0) Loss', 38.476509)], overall loss: -29.845718383789062
Iteration: 136, named_losses: [('ActivationMax Loss', -72.19326),
 ('L-6.0 Norm Loss', 0.062050804),
 ('TV(2.0) Loss', 39.395561)], overall loss: -32.73564910888672
Iteration: 137, named_losses: [('ActivationMax Loss', -68.521103),
 ('L-6.0 Norm Loss', 0.062057219),
 ('TV(2.0) Loss', 39.567799)], overall loss: -28.891246795654297
Iteration: 138, named_losses: [('ActivationMax Loss', -69.446297),
 ('L-6.0 Norm Loss', 0.062057234),
 ('TV(2.0) Loss', 39.070473)], overall loss: -30.313766479492188
Iteration: 139, named_losses: [('ActivationMax Loss', -68.955261),
 ('L-6.0 Norm Loss', 0.062060803),
 ('TV(2.0) Loss', 38.942894)], overall loss: -29.95030975341797
Iteration: 140, named_losses: [('ActivationMax Loss', -71.179245),
 ('L-6.0 Norm Loss', 0.062062152),
 ('TV(2.0) Loss', 39.347702)], overall loss: -31.76947784423828
Iteration: 141, named_losses: [('ActivationMax Loss', -69.470329),
 ('L-6.0 Norm Loss', 0.062066019),
 ('TV(2.0) Loss', 40.141434)], overall loss: -29.266830444335938
Iteration: 142, named_losses: [('ActivationMax Loss', -74.256622),
 ('L-6.0 Norm Loss', 0.062067755),
 ('TV(2.0) Loss', 40.202106)], overall loss: -33.99245071411133
Iteration: 143, named_losses: [('ActivationMax Loss', -70.539566),
 ('L-6.0 Norm Loss', 0.062071029),
 ('TV(2.0) Loss', 40.702305)], overall loss: -29.775188446044922
Iteration: 144, named_losses: [('ActivationMax Loss', -73.912994),
 ('L-6.0 Norm Loss', 0.062072743),
 ('TV(2.0) Loss', 40.235943)], overall loss: -33.6149787902832
Iteration: 145, named_losses: [('ActivationMax Loss', -71.829849),
 ('L-6.0 Norm Loss', 0.062075682),
 ('TV(2.0) Loss', 41.235737)], overall loss: -30.532039642333984
Iteration: 146, named_losses: [('ActivationMax Loss', -74.759377),
 ('L-6.0 Norm Loss', 0.062079821),
 ('TV(2.0) Loss', 40.284279)], overall loss: -34.41301727294922
Iteration: 147, named_losses: [('ActivationMax Loss', -73.852516),
 ('L-6.0 Norm Loss', 0.062081009),
 ('TV(2.0) Loss', 41.268486)], overall loss: -32.521949768066406
Iteration: 148, named_losses: [('ActivationMax Loss', -73.769432),
 ('L-6.0 Norm Loss', 0.062084191),
 ('TV(2.0) Loss', 41.814354)], overall loss: -31.89299774169922
Iteration: 149, named_losses: [('ActivationMax Loss', -73.207344),
 ('L-6.0 Norm Loss', 0.062085431),
 ('TV(2.0) Loss', 40.993202)], overall loss: -32.15205383300781
Iteration: 150, named_losses: [('ActivationMax Loss', -76.26609),
 ('L-6.0 Norm Loss', 0.062089007),
 ('TV(2.0) Loss', 41.458076)], overall loss: -34.74592590332031
Iteration: 151, named_losses: [('ActivationMax Loss', -72.275948),
 ('L-6.0 Norm Loss', 0.062090471),
 ('TV(2.0) Loss', 41.322922)], overall loss: -30.89093780517578
Iteration: 152, named_losses: [('ActivationMax Loss', -75.37191),
 ('L-6.0 Norm Loss', 0.062094301),
 ('TV(2.0) Loss', 40.983749)], overall loss: -34.32606506347656
Iteration: 153, named_losses: [('ActivationMax Loss', -74.64138),
 ('L-6.0 Norm Loss', 0.062096007),
 ('TV(2.0) Loss', 40.339821)], overall loss: -34.239463806152344
Iteration: 154, named_losses: [('ActivationMax Loss', -76.764885),
 ('L-6.0 Norm Loss', 0.062098481),
 ('TV(2.0) Loss', 40.326443)], overall loss: -36.376346588134766
Iteration: 155, named_losses: [('ActivationMax Loss', -73.65799),
 ('L-6.0 Norm Loss', 0.062101439),
 ('TV(2.0) Loss', 41.559887)], overall loss: -32.0359992980957
Iteration: 156, named_losses: [('ActivationMax Loss', -76.601265),
 ('L-6.0 Norm Loss', 0.062104236),
 ('TV(2.0) Loss', 40.871132)], overall loss: -35.66802978515625
Iteration: 157, named_losses: [('ActivationMax Loss', -72.490173),
 ('L-6.0 Norm Loss', 0.062106214),
 ('TV(2.0) Loss', 42.14415)], overall loss: -30.283920288085938
Iteration: 158, named_losses: [('ActivationMax Loss', -79.169991),
 ('L-6.0 Norm Loss', 0.062108692),
 ('TV(2.0) Loss', 41.277916)], overall loss: -37.82996368408203
Iteration: 159, named_losses: [('ActivationMax Loss', -74.585175),
 ('L-6.0 Norm Loss', 0.062111076),
 ('TV(2.0) Loss', 41.771225)], overall loss: -32.75183868408203
Iteration: 160, named_losses: [('ActivationMax Loss', -77.052193),
 ('L-6.0 Norm Loss', 0.062114276),
 ('TV(2.0) Loss', 42.5616)], overall loss: -34.42848205566406
Iteration: 161, named_losses: [('ActivationMax Loss', -76.438774),
 ('L-6.0 Norm Loss', 0.062115319),
 ('TV(2.0) Loss', 41.471237)], overall loss: -34.905418395996094
Iteration: 162, named_losses: [('ActivationMax Loss', -78.960793),
 ('L-6.0 Norm Loss', 0.062118489),
 ('TV(2.0) Loss', 43.069778)], overall loss: -35.828895568847656
Iteration: 163, named_losses: [('ActivationMax Loss', -73.679207),
 ('L-6.0 Norm Loss', 0.062120654),
 ('TV(2.0) Loss', 42.539005)], overall loss: -31.078083038330078
Iteration: 164, named_losses: [('ActivationMax Loss', -79.71209),
 ('L-6.0 Norm Loss', 0.062123515),
 ('TV(2.0) Loss', 43.494576)], overall loss: -36.15538787841797
Iteration: 165, named_losses: [('ActivationMax Loss', -77.443764),
 ('L-6.0 Norm Loss', 0.062124647),
 ('TV(2.0) Loss', 42.182899)], overall loss: -35.19873809814453
Iteration: 166, named_losses: [('ActivationMax Loss', -82.155434),
 ('L-6.0 Norm Loss', 0.062127855),
 ('TV(2.0) Loss', 42.845589)], overall loss: -39.247718811035156
Iteration: 167, named_losses: [('ActivationMax Loss', -77.442627),
 ('L-6.0 Norm Loss', 0.062130235),
 ('TV(2.0) Loss', 43.669277)], overall loss: -33.71121597290039
Iteration: 168, named_losses: [('ActivationMax Loss', -80.296135),
 ('L-6.0 Norm Loss', 0.062132262),
 ('TV(2.0) Loss', 43.704117)], overall loss: -36.529884338378906
Iteration: 169, named_losses: [('ActivationMax Loss', -79.159111),
 ('L-6.0 Norm Loss', 0.062134847),
 ('TV(2.0) Loss', 43.189545)], overall loss: -35.907432556152344
Iteration: 170, named_losses: [('ActivationMax Loss', -79.176773),
 ('L-6.0 Norm Loss', 0.062137157),
 ('TV(2.0) Loss', 44.845119)], overall loss: -34.2695198059082
Iteration: 171, named_losses: [('ActivationMax Loss', -76.232368),
 ('L-6.0 Norm Loss', 0.062142085),
 ('TV(2.0) Loss', 43.879807)], overall loss: -32.29042053222656
Iteration: 172, named_losses: [('ActivationMax Loss', -82.800385),
 ('L-6.0 Norm Loss', 0.062142),
 ('TV(2.0) Loss', 42.683388)], overall loss: -40.05485534667969
Iteration: 173, named_losses: [('ActivationMax Loss', -77.253883),
 ('L-6.0 Norm Loss', 0.062146075),
 ('TV(2.0) Loss', 44.798607)], overall loss: -32.39312744140625
Iteration: 174, named_losses: [('ActivationMax Loss', -82.107628),
 ('L-6.0 Norm Loss', 0.062146738),
 ('TV(2.0) Loss', 43.43008)], overall loss: -38.61539840698242
Iteration: 175, named_losses: [('ActivationMax Loss', -81.235428),
 ('L-6.0 Norm Loss', 0.06215091),
 ('TV(2.0) Loss', 44.47364)], overall loss: -36.69963836669922
Iteration: 176, named_losses: [('ActivationMax Loss', -81.216454),
 ('L-6.0 Norm Loss', 0.062152673),
 ('TV(2.0) Loss', 44.726379)], overall loss: -36.42792510986328
Iteration: 177, named_losses: [('ActivationMax Loss', -79.394073),
 ('L-6.0 Norm Loss', 0.06215626),
 ('TV(2.0) Loss', 43.223347)], overall loss: -36.10857009887695
Iteration: 178, named_losses: [('ActivationMax Loss', -81.420937),
 ('L-6.0 Norm Loss', 0.062156912),
 ('TV(2.0) Loss', 43.691936)], overall loss: -37.66684341430664
Iteration: 179, named_losses: [('ActivationMax Loss', -79.17379),
 ('L-6.0 Norm Loss', 0.062160004),
 ('TV(2.0) Loss', 43.258205)], overall loss: -35.85342788696289
Iteration: 180, named_losses: [('ActivationMax Loss', -83.363914),
 ('L-6.0 Norm Loss', 0.062161256),
 ('TV(2.0) Loss', 43.73616)], overall loss: -39.565589904785156
Iteration: 181, named_losses: [('ActivationMax Loss', -79.654846),
 ('L-6.0 Norm Loss', 0.062163997),
 ('TV(2.0) Loss', 43.423515)], overall loss: -36.169166564941406
Iteration: 182, named_losses: [('ActivationMax Loss', -82.149696),
 ('L-6.0 Norm Loss', 0.062166847),
 ('TV(2.0) Loss', 43.875198)], overall loss: -38.21233367919922
Iteration: 183, named_losses: [('ActivationMax Loss', -79.886551),
 ('L-6.0 Norm Loss', 0.062169638),
 ('TV(2.0) Loss', 43.191624)], overall loss: -36.632755279541016
Iteration: 184, named_losses: [('ActivationMax Loss', -83.084366),
 ('L-6.0 Norm Loss', 0.062172841),
 ('TV(2.0) Loss', 43.547802)], overall loss: -39.47439193725586
Iteration: 185, named_losses: [('ActivationMax Loss', -82.260071),
 ('L-6.0 Norm Loss', 0.062175244),
 ('TV(2.0) Loss', 44.121773)], overall loss: -38.07612609863281
Iteration: 186, named_losses: [('ActivationMax Loss', -85.2985),
 ('L-6.0 Norm Loss', 0.062178183),
 ('TV(2.0) Loss', 45.06609)], overall loss: -40.170230865478516
Iteration: 187, named_losses: [('ActivationMax Loss', -80.46936),
 ('L-6.0 Norm Loss', 0.062181078),
 ('TV(2.0) Loss', 44.712605)], overall loss: -35.694576263427734
Iteration: 188, named_losses: [('ActivationMax Loss', -83.824646),
 ('L-6.0 Norm Loss', 0.062185075),
 ('TV(2.0) Loss', 45.65036)], overall loss: -38.112098693847656
Iteration: 189, named_losses: [('ActivationMax Loss', -83.467606),
 ('L-6.0 Norm Loss', 0.06218582),
 ('TV(2.0) Loss', 44.686069)], overall loss: -38.7193489074707
Iteration: 190, named_losses: [('ActivationMax Loss', -84.42411),
 ('L-6.0 Norm Loss', 0.062189694),
 ('TV(2.0) Loss', 46.027153)], overall loss: -38.33477020263672
Iteration: 191, named_losses: [('ActivationMax Loss', -82.170158),
 ('L-6.0 Norm Loss', 0.062191021),
 ('TV(2.0) Loss', 45.756798)], overall loss: -36.351165771484375
Iteration: 192, named_losses: [('ActivationMax Loss', -86.53582),
 ('L-6.0 Norm Loss', 0.06219507),
 ('TV(2.0) Loss', 45.986153)], overall loss: -40.48747253417969
Iteration: 193, named_losses: [('ActivationMax Loss', -81.639473),
 ('L-6.0 Norm Loss', 0.062196828),
 ('TV(2.0) Loss', 46.255779)], overall loss: -35.32149887084961
Iteration: 194, named_losses: [('ActivationMax Loss', -83.569382),
 ('L-6.0 Norm Loss', 0.062200617),
 ('TV(2.0) Loss', 45.552719)], overall loss: -37.95446014404297
Iteration: 195, named_losses: [('ActivationMax Loss', -86.916046),
 ('L-6.0 Norm Loss', 0.062201425),
 ('TV(2.0) Loss', 45.176758)], overall loss: -41.677085876464844
Iteration: 196, named_losses: [('ActivationMax Loss', -84.032524),
 ('L-6.0 Norm Loss', 0.062204879),
 ('TV(2.0) Loss', 46.132248)], overall loss: -37.83807373046875
Iteration: 197, named_losses: [('ActivationMax Loss', -85.557144),
 ('L-6.0 Norm Loss', 0.062207341),
 ('TV(2.0) Loss', 46.082672)], overall loss: -39.412261962890625
Iteration: 198, named_losses: [('ActivationMax Loss', -83.613594),
 ('L-6.0 Norm Loss', 0.062210388),
 ('TV(2.0) Loss', 46.568359)], overall loss: -36.98302459716797
Iteration: 199, named_losses: [('ActivationMax Loss', -86.791801),
 ('L-6.0 Norm Loss', 0.062212639),
 ('TV(2.0) Loss', 45.134106)], overall loss: -41.59548568725586
Iteration: 200, named_losses: [('ActivationMax Loss', -81.647064),
 ('L-6.0 Norm Loss', 0.062215641),
 ('TV(2.0) Loss', 45.751949)], overall loss: -35.8328971862793
Iteration: 201, named_losses: [('ActivationMax Loss', -87.816864),
 ('L-6.0 Norm Loss', 0.062217705),
 ('TV(2.0) Loss', 45.276299)], overall loss: -42.47834777832031
Iteration: 202, named_losses: [('ActivationMax Loss', -84.383942),
 ('L-6.0 Norm Loss', 0.062220536),
 ('TV(2.0) Loss', 46.618649)], overall loss: -37.70307540893555
Iteration: 203, named_losses: [('ActivationMax Loss', -87.794357),
 ('L-6.0 Norm Loss', 0.06222181),
 ('TV(2.0) Loss', 45.836372)], overall loss: -41.89575958251953
Iteration: 204, named_losses: [('ActivationMax Loss', -88.159592),
 ('L-6.0 Norm Loss', 0.062225409),
 ('TV(2.0) Loss', 47.578583)], overall loss: -40.51878356933594
Iteration: 205, named_losses: [('ActivationMax Loss', -88.599762),
 ('L-6.0 Norm Loss', 0.062227972),
 ('TV(2.0) Loss', 47.090401)], overall loss: -41.44713592529297
Iteration: 206, named_losses: [('ActivationMax Loss', -88.90831),
 ('L-6.0 Norm Loss', 0.062229931),
 ('TV(2.0) Loss', 47.677921)], overall loss: -41.168155670166016
Iteration: 207, named_losses: [('ActivationMax Loss', -87.245583),
 ('L-6.0 Norm Loss', 0.062234111),
 ('TV(2.0) Loss', 46.405636)], overall loss: -40.777713775634766
Iteration: 208, named_losses: [('ActivationMax Loss', -84.693764),
 ('L-6.0 Norm Loss', 0.062235408),
 ('TV(2.0) Loss', 47.227322)], overall loss: -37.40420913696289
Iteration: 209, named_losses: [('ActivationMax Loss', -85.826042),
 ('L-6.0 Norm Loss', 0.062238466),
 ('TV(2.0) Loss', 44.977081)], overall loss: -40.786720275878906
Iteration: 210, named_losses: [('ActivationMax Loss', -89.670929),
 ('L-6.0 Norm Loss', 0.062239155),
 ('TV(2.0) Loss', 45.95871)], overall loss: -43.64997863769531
Iteration: 211, named_losses: [('ActivationMax Loss', -83.174995),
 ('L-6.0 Norm Loss', 0.062243581),
 ('TV(2.0) Loss', 46.175114)], overall loss: -36.93764114379883
Iteration: 212, named_losses: [('ActivationMax Loss', -87.912514),
 ('L-6.0 Norm Loss', 0.062245309),
 ('TV(2.0) Loss', 46.234467)], overall loss: -41.61579895019531
Iteration: 213, named_losses: [('ActivationMax Loss', -85.537399),
 ('L-6.0 Norm Loss', 0.062247995),
 ('TV(2.0) Loss', 45.856003)], overall loss: -39.61914825439453
Iteration: 214, named_losses: [('ActivationMax Loss', -89.612312),
 ('L-6.0 Norm Loss', 0.062250495),
 ('TV(2.0) Loss', 46.025375)], overall loss: -43.524688720703125
Iteration: 215, named_losses: [('ActivationMax Loss', -84.610123),
 ('L-6.0 Norm Loss', 0.062253393),
 ('TV(2.0) Loss', 46.508888)], overall loss: -38.038978576660156
Iteration: 216, named_losses: [('ActivationMax Loss', -90.493111),
 ('L-6.0 Norm Loss', 0.062255096),
 ('TV(2.0) Loss', 46.15316)], overall loss: -44.27769470214844
Iteration: 217, named_losses: [('ActivationMax Loss', -86.011971),
 ('L-6.0 Norm Loss', 0.062257968),
 ('TV(2.0) Loss', 46.398102)], overall loss: -39.551612854003906
Iteration: 218, named_losses: [('ActivationMax Loss', -89.159111),
 ('L-6.0 Norm Loss', 0.06225998),
 ('TV(2.0) Loss', 46.619118)], overall loss: -42.47772979736328
Iteration: 219, named_losses: [('ActivationMax Loss', -86.063301),
 ('L-6.0 Norm Loss', 0.062261604),
 ('TV(2.0) Loss', 46.419273)], overall loss: -39.581764221191406
Iteration: 220, named_losses: [('ActivationMax Loss', -87.915474),
 ('L-6.0 Norm Loss', 0.062266406),
 ('TV(2.0) Loss', 46.782974)], overall loss: -41.07023620605469
Iteration: 221, named_losses: [('ActivationMax Loss', -88.851227),
 ('L-6.0 Norm Loss', 0.062266394),
 ('TV(2.0) Loss', 46.57296)], overall loss: -42.21600341796875
Iteration: 222, named_losses: [('ActivationMax Loss', -89.184006),
 ('L-6.0 Norm Loss', 0.062271468),
 ('TV(2.0) Loss', 47.125973)], overall loss: -41.99576187133789
Iteration: 223, named_losses: [('ActivationMax Loss', -89.636177),
 ('L-6.0 Norm Loss', 0.062271826),
 ('TV(2.0) Loss', 47.010056)], overall loss: -42.56385040283203
Iteration: 224, named_losses: [('ActivationMax Loss', -90.784798),
 ('L-6.0 Norm Loss', 0.06227741),
 ('TV(2.0) Loss', 47.700699)], overall loss: -43.021820068359375
Iteration: 225, named_losses: [('ActivationMax Loss', -89.625015),
 ('L-6.0 Norm Loss', 0.062279008),
 ('TV(2.0) Loss', 48.092655)], overall loss: -41.4700813293457
Iteration: 226, named_losses: [('ActivationMax Loss', -91.009491),
 ('L-6.0 Norm Loss', 0.062281571),
 ('TV(2.0) Loss', 48.038906)], overall loss: -42.90830612182617
Iteration: 227, named_losses: [('ActivationMax Loss', -87.907745),
 ('L-6.0 Norm Loss', 0.062284358),
 ('TV(2.0) Loss', 48.678986)], overall loss: -39.166473388671875
Iteration: 228, named_losses: [('ActivationMax Loss', -92.290337),
 ('L-6.0 Norm Loss', 0.06228698),
 ('TV(2.0) Loss', 48.364529)], overall loss: -43.863521575927734
Iteration: 229, named_losses: [('ActivationMax Loss', -87.596916),
 ('L-6.0 Norm Loss', 0.062290318),
 ('TV(2.0) Loss', 49.230019)], overall loss: -38.304603576660156
Iteration: 230, named_losses: [('ActivationMax Loss', -92.716133),
 ('L-6.0 Norm Loss', 0.062293191),
 ('TV(2.0) Loss', 49.121307)], overall loss: -43.53253173828125
Iteration: 231, named_losses: [('ActivationMax Loss', -88.449654),
 ('L-6.0 Norm Loss', 0.062295035),
 ('TV(2.0) Loss', 48.285137)], overall loss: -40.10222244262695
Iteration: 232, named_losses: [('ActivationMax Loss', -93.457291),
 ('L-6.0 Norm Loss', 0.062297441),
 ('TV(2.0) Loss', 48.001976)], overall loss: -45.39302062988281
Iteration: 233, named_losses: [('ActivationMax Loss', -88.755898),
 ('L-6.0 Norm Loss', 0.062299922),
 ('TV(2.0) Loss', 49.707943)], overall loss: -38.985652923583984
Iteration: 234, named_losses: [('ActivationMax Loss', -92.334045),
 ('L-6.0 Norm Loss', 0.062302485),
 ('TV(2.0) Loss', 49.45649)], overall loss: -42.81525421142578
Iteration: 235, named_losses: [('ActivationMax Loss', -90.230087),
 ('L-6.0 Norm Loss', 0.062304683),
 ('TV(2.0) Loss', 48.900146)], overall loss: -41.26763916015625
Iteration: 236, named_losses: [('ActivationMax Loss', -95.343925),
 ('L-6.0 Norm Loss', 0.062307011),
 ('TV(2.0) Loss', 49.529972)], overall loss: -45.751644134521484
Iteration: 237, named_losses: [('ActivationMax Loss', -90.401344),
 ('L-6.0 Norm Loss', 0.062309727),
 ('TV(2.0) Loss', 50.37825)], overall loss: -39.960784912109375
Iteration: 238, named_losses: [('ActivationMax Loss', -94.933884),
 ('L-6.0 Norm Loss', 0.062311843),
 ('TV(2.0) Loss', 49.541519)], overall loss: -45.330055236816406
Iteration: 239, named_losses: [('ActivationMax Loss', -90.336517),
 ('L-6.0 Norm Loss', 0.062313534),
 ('TV(2.0) Loss', 49.67535)], overall loss: -40.59885025024414
Iteration: 240, named_losses: [('ActivationMax Loss', -92.478806),
 ('L-6.0 Norm Loss', 0.062316246),
 ('TV(2.0) Loss', 48.575035)], overall loss: -43.841453552246094
Iteration: 241, named_losses: [('ActivationMax Loss', -92.967834),
 ('L-6.0 Norm Loss', 0.06231726),
 ('TV(2.0) Loss', 49.731895)], overall loss: -43.173622131347656
Iteration: 242, named_losses: [('ActivationMax Loss', -89.102119),
 ('L-6.0 Norm Loss', 0.062321652),
 ('TV(2.0) Loss', 48.416309)], overall loss: -40.62348556518555
Iteration: 243, named_losses: [('ActivationMax Loss', -93.214195),
 ('L-6.0 Norm Loss', 0.06232173),
 ('TV(2.0) Loss', 48.786678)], overall loss: -44.36519241333008
Iteration: 244, named_losses: [('ActivationMax Loss', -92.252098),
 ('L-6.0 Norm Loss', 0.062327355),
 ('TV(2.0) Loss', 47.914391)], overall loss: -44.27538299560547
Iteration: 245, named_losses: [('ActivationMax Loss', -93.325478),
 ('L-6.0 Norm Loss', 0.062328134),
 ('TV(2.0) Loss', 49.730217)], overall loss: -43.532936096191406
Iteration: 246, named_losses: [('ActivationMax Loss', -92.493515),
 ('L-6.0 Norm Loss', 0.062331751),
 ('TV(2.0) Loss', 48.478184)], overall loss: -43.952999114990234
Iteration: 247, named_losses: [('ActivationMax Loss', -95.675095),
 ('L-6.0 Norm Loss', 0.062332533),
 ('TV(2.0) Loss', 49.523918)], overall loss: -46.088844299316406
Iteration: 248, named_losses: [('ActivationMax Loss', -90.225891),
 ('L-6.0 Norm Loss', 0.06233684),
 ('TV(2.0) Loss', 50.325432)], overall loss: -39.83811950683594
Iteration: 249, named_losses: [('ActivationMax Loss', -97.693451),
 ('L-6.0 Norm Loss', 0.062337186),
 ('TV(2.0) Loss', 49.690567)], overall loss: -47.94054412841797
Iteration: 250, named_losses: [('ActivationMax Loss', -91.265923),
 ('L-6.0 Norm Loss', 0.062341452),
 ('TV(2.0) Loss', 51.236557)], overall loss: -39.96702575683594
Iteration: 251, named_losses: [('ActivationMax Loss', -95.060417),
 ('L-6.0 Norm Loss', 0.062342476),
 ('TV(2.0) Loss', 49.996887)], overall loss: -45.001190185546875
Iteration: 252, named_losses: [('ActivationMax Loss', -89.965355),
 ('L-6.0 Norm Loss', 0.062344443),
 ('TV(2.0) Loss', 49.623409)], overall loss: -40.279598236083984
Iteration: 253, named_losses: [('ActivationMax Loss', -94.512901),
 ('L-6.0 Norm Loss', 0.062347665),
 ('TV(2.0) Loss', 49.581512)], overall loss: -44.869041442871094
Iteration: 254, named_losses: [('ActivationMax Loss', -91.515976),
 ('L-6.0 Norm Loss', 0.062349256),
 ('TV(2.0) Loss', 48.764591)], overall loss: -42.68903732299805
Iteration: 255, named_losses: [('ActivationMax Loss', -93.680359),
 ('L-6.0 Norm Loss', 0.062353209),
 ('TV(2.0) Loss', 48.607632)], overall loss: -45.010372161865234
Iteration: 256, named_losses: [('ActivationMax Loss', -92.872772),
 ('L-6.0 Norm Loss', 0.062354352),
 ('TV(2.0) Loss', 48.47126)], overall loss: -44.33915710449219
Iteration: 257, named_losses: [('ActivationMax Loss', -95.5933),
 ('L-6.0 Norm Loss', 0.06235832),
 ('TV(2.0) Loss', 49.241982)], overall loss: -46.288963317871094
Iteration: 258, named_losses: [('ActivationMax Loss', -92.063339),
 ('L-6.0 Norm Loss', 0.062359728),
 ('TV(2.0) Loss', 48.584732)], overall loss: -43.41624450683594
Iteration: 259, named_losses: [('ActivationMax Loss', -97.811737),
 ('L-6.0 Norm Loss', 0.062363151),
 ('TV(2.0) Loss', 48.736687)], overall loss: -49.01268768310547
Iteration: 260, named_losses: [('ActivationMax Loss', -92.474609),
 ('L-6.0 Norm Loss', 0.062363401),
 ('TV(2.0) Loss', 49.452415)], overall loss: -42.95983123779297
Iteration: 261, named_losses: [('ActivationMax Loss', -96.649292),
 ('L-6.0 Norm Loss', 0.062367376),
 ('TV(2.0) Loss', 49.6008)], overall loss: -46.986122131347656
Iteration: 262, named_losses: [('ActivationMax Loss', -93.698441),
 ('L-6.0 Norm Loss', 0.062369131),
 ('TV(2.0) Loss', 50.219006)], overall loss: -43.41706466674805
Iteration: 263, named_losses: [('ActivationMax Loss', -96.923912),
 ('L-6.0 Norm Loss', 0.062371846),
 ('TV(2.0) Loss', 49.774899)], overall loss: -47.08664321899414
Iteration: 264, named_losses: [('ActivationMax Loss', -94.612488),
 ('L-6.0 Norm Loss', 0.062373608),
 ('TV(2.0) Loss', 51.515118)], overall loss: -43.03499984741211
Iteration: 265, named_losses: [('ActivationMax Loss', -96.170403),
 ('L-6.0 Norm Loss', 0.062376559),
 ('TV(2.0) Loss', 50.919388)], overall loss: -45.188636779785156
Iteration: 266, named_losses: [('ActivationMax Loss', -94.197289),
 ('L-6.0 Norm Loss', 0.062378332),
 ('TV(2.0) Loss', 51.59185)], overall loss: -42.543060302734375
Iteration: 267, named_losses: [('ActivationMax Loss', -96.970078),
 ('L-6.0 Norm Loss', 0.062379852),
 ('TV(2.0) Loss', 50.977253)], overall loss: -45.93044662475586
Iteration: 268, named_losses: [('ActivationMax Loss', -96.338783),
 ('L-6.0 Norm Loss', 0.062383559),
 ('TV(2.0) Loss', 52.29908)], overall loss: -43.977317810058594
Iteration: 269, named_losses: [('ActivationMax Loss', -98.555267),
 ('L-6.0 Norm Loss', 0.062385421),
 ('TV(2.0) Loss', 51.284481)], overall loss: -47.20840072631836
Iteration: 270, named_losses: [('ActivationMax Loss', -94.628601),
 ('L-6.0 Norm Loss', 0.062389053),
 ('TV(2.0) Loss', 52.267899)], overall loss: -42.298316955566406
Iteration: 271, named_losses: [('ActivationMax Loss', -100.11405),
 ('L-6.0 Norm Loss', 0.062390685),
 ('TV(2.0) Loss', 50.978378)], overall loss: -49.073280334472656
Iteration: 272, named_losses: [('ActivationMax Loss', -93.630562),
 ('L-6.0 Norm Loss', 0.062392563),
 ('TV(2.0) Loss', 50.697739)], overall loss: -42.87042999267578
Iteration: 273, named_losses: [('ActivationMax Loss', -100.56676),
 ('L-6.0 Norm Loss', 0.062395483),
 ('TV(2.0) Loss', 51.188629)], overall loss: -49.31573486328125
Iteration: 274, named_losses: [('ActivationMax Loss', -91.548218),
 ('L-6.0 Norm Loss', 0.062397253),
 ('TV(2.0) Loss', 50.62307)], overall loss: -40.86274719238281
Iteration: 275, named_losses: [('ActivationMax Loss', -96.794365),
 ('L-6.0 Norm Loss', 0.062400743),
 ('TV(2.0) Loss', 50.752022)], overall loss: -45.979942321777344
Iteration: 276, named_losses: [('ActivationMax Loss', -94.68708),
 ('L-6.0 Norm Loss', 0.062401533),
 ('TV(2.0) Loss', 49.174561)], overall loss: -45.45011901855469
Iteration: 277, named_losses: [('ActivationMax Loss', -98.390915),
 ('L-6.0 Norm Loss', 0.062406249),
 ('TV(2.0) Loss', 50.733315)], overall loss: -47.595191955566406
Iteration: 278, named_losses: [('ActivationMax Loss', -96.74987),
 ('L-6.0 Norm Loss', 0.062406093),
 ('TV(2.0) Loss', 49.42255)], overall loss: -47.26491165161133
Iteration: 279, named_losses: [('ActivationMax Loss', -97.804291),
 ('L-6.0 Norm Loss', 0.062410515),
 ('TV(2.0) Loss', 50.034313)], overall loss: -47.70756912231445
Iteration: 280, named_losses: [('ActivationMax Loss', -95.013451),
 ('L-6.0 Norm Loss', 0.062411517),
 ('TV(2.0) Loss', 49.645573)], overall loss: -45.30546951293945
Iteration: 281, named_losses: [('ActivationMax Loss', -99.008118),
 ('L-6.0 Norm Loss', 0.062415678),
 ('TV(2.0) Loss', 50.874271)], overall loss: -48.07143020629883
Iteration: 282, named_losses: [('ActivationMax Loss', -95.196724),
 ('L-6.0 Norm Loss', 0.062415637),
 ('TV(2.0) Loss', 50.178413)], overall loss: -44.955894470214844
Iteration: 283, named_losses: [('ActivationMax Loss', -96.757927),
 ('L-6.0 Norm Loss', 0.06242267),
 ('TV(2.0) Loss', 52.127693)], overall loss: -44.56781005859375
Iteration: 284, named_losses: [('ActivationMax Loss', -96.785004),
 ('L-6.0 Norm Loss', 0.062423207),
 ('TV(2.0) Loss', 51.707855)], overall loss: -45.01472473144531
Iteration: 285, named_losses: [('ActivationMax Loss', -95.348557),
 ('L-6.0 Norm Loss', 0.062426344),
 ('TV(2.0) Loss', 52.688866)], overall loss: -42.597267150878906
Iteration: 286, named_losses: [('ActivationMax Loss', -101.2824),
 ('L-6.0 Norm Loss', 0.062425505),
 ('TV(2.0) Loss', 51.014229)], overall loss: -50.20574951171875
Iteration: 287, named_losses: [('ActivationMax Loss', -94.734062),
 ('L-6.0 Norm Loss', 0.062430419),
 ('TV(2.0) Loss', 51.780998)], overall loss: -42.89063262939453
Iteration: 288, named_losses: [('ActivationMax Loss', -99.572952),
 ('L-6.0 Norm Loss', 0.06243141),
 ('TV(2.0) Loss', 52.298134)], overall loss: -47.21238708496094
Iteration: 289, named_losses: [('ActivationMax Loss', -97.005966),
 ('L-6.0 Norm Loss', 0.062433451),
 ('TV(2.0) Loss', 52.471127)], overall loss: -44.472408294677734
Iteration: 290, named_losses: [('ActivationMax Loss', -100.72915),
 ('L-6.0 Norm Loss', 0.062435575),
 ('TV(2.0) Loss', 53.236023)], overall loss: -47.430686950683594
Iteration: 291, named_losses: [('ActivationMax Loss', -97.2892),
 ('L-6.0 Norm Loss', 0.062438332),
 ('TV(2.0) Loss', 52.686073)], overall loss: -44.540687561035156
Iteration: 292, named_losses: [('ActivationMax Loss', -101.23533),
 ('L-6.0 Norm Loss', 0.06244019),
 ('TV(2.0) Loss', 53.578178)], overall loss: -47.59471130371094
Iteration: 293, named_losses: [('ActivationMax Loss', -92.900848),
 ('L-6.0 Norm Loss', 0.062444694),
 ('TV(2.0) Loss', 51.992119)], overall loss: -40.846282958984375
Iteration: 294, named_losses: [('ActivationMax Loss', -103.37528),
 ('L-6.0 Norm Loss', 0.062446035),
 ('TV(2.0) Loss', 51.712189)], overall loss: -51.60064697265625
Iteration: 295, named_losses: [('ActivationMax Loss', -95.680206),
 ('L-6.0 Norm Loss', 0.062449355),
 ('TV(2.0) Loss', 51.723904)], overall loss: -43.893856048583984
Iteration: 296, named_losses: [('ActivationMax Loss', -102.40531),
 ('L-6.0 Norm Loss', 0.062450118),
 ('TV(2.0) Loss', 50.64769)], overall loss: -51.69517517089844
Iteration: 297, named_losses: [('ActivationMax Loss', -98.976929),
 ('L-6.0 Norm Loss', 0.062453449),
 ('TV(2.0) Loss', 52.072464)], overall loss: -46.842010498046875
Iteration: 298, named_losses: [('ActivationMax Loss', -100.78832),
 ('L-6.0 Norm Loss', 0.062454533),
 ('TV(2.0) Loss', 51.930828)], overall loss: -48.795040130615234
Iteration: 299, named_losses: [('ActivationMax Loss', -96.741692),
 ('L-6.0 Norm Loss', 0.062457897),
 ('TV(2.0) Loss', 51.468834)], overall loss: -45.21040344238281
Iteration: 300, named_losses: [('ActivationMax Loss', -102.04729),
 ('L-6.0 Norm Loss', 0.062459745),
 ('TV(2.0) Loss', 50.366482)], overall loss: -51.618343353271484
Iteration: 301, named_losses: [('ActivationMax Loss', -97.041908),
 ('L-6.0 Norm Loss', 0.062461123),
 ('TV(2.0) Loss', 51.089325)], overall loss: -45.89012145996094
Iteration: 302, named_losses: [('ActivationMax Loss', -100.96075),
 ('L-6.0 Norm Loss', 0.062464595),
 ('TV(2.0) Loss', 51.016479)], overall loss: -49.881813049316406
Iteration: 303, named_losses: [('ActivationMax Loss', -97.390915),
 ('L-6.0 Norm Loss', 0.062467217),
 ('TV(2.0) Loss', 51.244896)], overall loss: -46.08354949951172
Iteration: 304, named_losses: [('ActivationMax Loss', -101.27479),
 ('L-6.0 Norm Loss', 0.062470376),
 ('TV(2.0) Loss', 50.154388)], overall loss: -51.05792999267578
Iteration: 305, named_losses: [('ActivationMax Loss', -98.859833),
 ('L-6.0 Norm Loss', 0.062471345),
 ('TV(2.0) Loss', 50.71236)], overall loss: -48.08500289916992
Iteration: 306, named_losses: [('ActivationMax Loss', -97.486641),
 ('L-6.0 Norm Loss', 0.062475123),
 ('TV(2.0) Loss', 51.022678)], overall loss: -46.401485443115234
Iteration: 307, named_losses: [('ActivationMax Loss', -103.04443),
 ('L-6.0 Norm Loss', 0.062475815),
 ('TV(2.0) Loss', 51.620007)], overall loss: -51.3619499206543
Iteration: 308, named_losses: [('ActivationMax Loss', -96.99247),
 ('L-6.0 Norm Loss', 0.062479444),
 ('TV(2.0) Loss', 51.668747)], overall loss: -45.26124572753906
Iteration: 309, named_losses: [('ActivationMax Loss', -103.48102),
 ('L-6.0 Norm Loss', 0.062480271),
 ('TV(2.0) Loss', 52.375141)], overall loss: -51.043399810791016
Iteration: 310, named_losses: [('ActivationMax Loss', -100.02702),
 ('L-6.0 Norm Loss', 0.062483422),
 ('TV(2.0) Loss', 52.342686)], overall loss: -47.62184524536133
Iteration: 311, named_losses: [('ActivationMax Loss', -104.43633),
 ('L-6.0 Norm Loss', 0.062484819),
 ('TV(2.0) Loss', 53.61866)], overall loss: -50.75518035888672
Iteration: 312, named_losses: [('ActivationMax Loss', -98.684212),
 ('L-6.0 Norm Loss', 0.062485617),
 ('TV(2.0) Loss', 53.858841)], overall loss: -44.76288604736328
Iteration: 313, named_losses: [('ActivationMax Loss', -103.0603),
 ('L-6.0 Norm Loss', 0.062489305),
 ('TV(2.0) Loss', 53.547676)], overall loss: -49.45012664794922
Iteration: 314, named_losses: [('ActivationMax Loss', -100.98482),
 ('L-6.0 Norm Loss', 0.06248999),
 ('TV(2.0) Loss', 52.794525)], overall loss: -48.12779998779297
Iteration: 315, named_losses: [('ActivationMax Loss', -103.10455),
 ('L-6.0 Norm Loss', 0.062492732),
 ('TV(2.0) Loss', 53.725159)], overall loss: -49.31689453125
Iteration: 316, named_losses: [('ActivationMax Loss', -99.131218),
 ('L-6.0 Norm Loss', 0.062494982),
 ('TV(2.0) Loss', 52.780952)], overall loss: -46.28777313232422
Iteration: 317, named_losses: [('ActivationMax Loss', -102.30113),
 ('L-6.0 Norm Loss', 0.062498763),
 ('TV(2.0) Loss', 52.874001)], overall loss: -49.36463165283203
Iteration: 318, named_losses: [('ActivationMax Loss', -102.99293),
 ('L-6.0 Norm Loss', 0.062499527),
 ('TV(2.0) Loss', 51.825783)], overall loss: -51.104644775390625
Iteration: 319, named_losses: [('ActivationMax Loss', -99.057594),
 ('L-6.0 Norm Loss', 0.062504523),
 ('TV(2.0) Loss', 52.571953)], overall loss: -46.423133850097656
Iteration: 320, named_losses: [('ActivationMax Loss', -101.44371),
 ('L-6.0 Norm Loss', 0.062505081),
 ('TV(2.0) Loss', 52.326744)], overall loss: -49.05445861816406
Iteration: 321, named_losses: [('ActivationMax Loss', -101.81977),
 ('L-6.0 Norm Loss', 0.062509529),
 ('TV(2.0) Loss', 51.408459)], overall loss: -50.34880447387695
Iteration: 322, named_losses: [('ActivationMax Loss', -100.63535),
 ('L-6.0 Norm Loss', 0.062510364),
 ('TV(2.0) Loss', 52.494888)], overall loss: -48.07794952392578
Iteration: 323, named_losses: [('ActivationMax Loss', -99.690681),
 ('L-6.0 Norm Loss', 0.062514901),
 ('TV(2.0) Loss', 51.735912)], overall loss: -47.89225387573242
Iteration: 324, named_losses: [('ActivationMax Loss', -101.55682),
 ('L-6.0 Norm Loss', 0.062516063),
 ('TV(2.0) Loss', 51.734333)], overall loss: -49.75996780395508
Iteration: 325, named_losses: [('ActivationMax Loss', -101.75683),
 ('L-6.0 Norm Loss', 0.062520482),
 ('TV(2.0) Loss', 51.638191)], overall loss: -50.056114196777344
Iteration: 326, named_losses: [('ActivationMax Loss', -97.312721),
 ('L-6.0 Norm Loss', 0.062521636),
 ('TV(2.0) Loss', 52.275772)], overall loss: -44.97442626953125
Iteration: 327, named_losses: [('ActivationMax Loss', -103.6657),
 ('L-6.0 Norm Loss', 0.062523685),
 ('TV(2.0) Loss', 51.910439)], overall loss: -51.69274139404297
Iteration: 328, named_losses: [('ActivationMax Loss', -97.726349),
 ('L-6.0 Norm Loss', 0.062524676),
 ('TV(2.0) Loss', 51.789707)], overall loss: -45.87411880493164
Iteration: 329, named_losses: [('ActivationMax Loss', -104.54907),
 ('L-6.0 Norm Loss', 0.062528268),
 ('TV(2.0) Loss', 52.560509)], overall loss: -51.92603302001953
Iteration: 330, named_losses: [('ActivationMax Loss', -97.6688),
 ('L-6.0 Norm Loss', 0.062529534),
 ('TV(2.0) Loss', 52.751343)], overall loss: -44.85492706298828
Iteration: 331, named_losses: [('ActivationMax Loss', -103.49584),
 ('L-6.0 Norm Loss', 0.062533036),
 ('TV(2.0) Loss', 52.716827)], overall loss: -50.71648406982422
Iteration: 332, named_losses: [('ActivationMax Loss', -99.051048),
 ('L-6.0 Norm Loss', 0.062533177),
 ('TV(2.0) Loss', 53.373981)], overall loss: -45.61453628540039
Iteration: 333, named_losses: [('ActivationMax Loss', -98.644051),
 ('L-6.0 Norm Loss', 0.062538058),
 ('TV(2.0) Loss', 52.556919)], overall loss: -46.024593353271484
Iteration: 334, named_losses: [('ActivationMax Loss', -104.12374),
 ('L-6.0 Norm Loss', 0.062539309),
 ('TV(2.0) Loss', 53.237122)], overall loss: -50.82408142089844
Iteration: 335, named_losses: [('ActivationMax Loss', -99.745964),
 ('L-6.0 Norm Loss', 0.062542915),
 ('TV(2.0) Loss', 53.52356)], overall loss: -46.15985870361328
Iteration: 336, named_losses: [('ActivationMax Loss', -105.0116),
 ('L-6.0 Norm Loss', 0.062544882),
 ('TV(2.0) Loss', 54.201546)], overall loss: -50.74750518798828
Iteration: 337, named_losses: [('ActivationMax Loss', -101.50762),
 ('L-6.0 Norm Loss', 0.062548697),
 ('TV(2.0) Loss', 53.969597)], overall loss: -47.47547912597656
Iteration: 338, named_losses: [('ActivationMax Loss', -106.18888),
 ('L-6.0 Norm Loss', 0.062548891),
 ('TV(2.0) Loss', 54.76442)], overall loss: -51.361915588378906
Iteration: 339, named_losses: [('ActivationMax Loss', -103.67889),
 ('L-6.0 Norm Loss', 0.062552311),
 ('TV(2.0) Loss', 54.801029)], overall loss: -48.815311431884766
Iteration: 340, named_losses: [('ActivationMax Loss', -104.76196),
 ('L-6.0 Norm Loss', 0.062554114),
 ('TV(2.0) Loss', 55.20863)], overall loss: -49.49077224731445
Iteration: 341, named_losses: [('ActivationMax Loss', -104.36144),
 ('L-6.0 Norm Loss', 0.062557034),
 ('TV(2.0) Loss', 53.863377)], overall loss: -50.43551254272461
Iteration: 342, named_losses: [('ActivationMax Loss', -106.48806),
 ('L-6.0 Norm Loss', 0.062558815),
 ('TV(2.0) Loss', 54.560356)], overall loss: -51.865142822265625
Iteration: 343, named_losses: [('ActivationMax Loss', -102.40833),
 ('L-6.0 Norm Loss', 0.06256175),
 ('TV(2.0) Loss', 55.086403)], overall loss: -47.259361267089844
Iteration: 344, named_losses: [('ActivationMax Loss', -107.57236),
 ('L-6.0 Norm Loss', 0.062562026),
 ('TV(2.0) Loss', 54.135265)], overall loss: -53.37453842163086
Iteration: 345, named_losses: [('ActivationMax Loss', -99.675209),
 ('L-6.0 Norm Loss', 0.062565193),
 ('TV(2.0) Loss', 54.129189)], overall loss: -45.48345184326172
Iteration: 346, named_losses: [('ActivationMax Loss', -107.13038),
 ('L-6.0 Norm Loss', 0.062566906),
 ('TV(2.0) Loss', 53.474331)], overall loss: -53.59347915649414
Iteration: 347, named_losses: [('ActivationMax Loss', -96.551025),
 ('L-6.0 Norm Loss', 0.062569842),
 ('TV(2.0) Loss', 53.64016)], overall loss: -42.848297119140625
Iteration: 348, named_losses: [('ActivationMax Loss', -108.51188),
 ('L-6.0 Norm Loss', 0.062571473),
 ('TV(2.0) Loss', 52.741886)], overall loss: -55.70742416381836
Iteration: 349, named_losses: [('ActivationMax Loss', -98.216972),
 ('L-6.0 Norm Loss', 0.062572911),
 ('TV(2.0) Loss', 52.137676)], overall loss: -46.016719818115234
Iteration: 350, named_losses: [('ActivationMax Loss', -106.3744),
 ('L-6.0 Norm Loss', 0.062576391),
 ('TV(2.0) Loss', 52.997231)], overall loss: -53.31459045410156
Iteration: 351, named_losses: [('ActivationMax Loss', -100.96413),
 ('L-6.0 Norm Loss', 0.062578529),
 ('TV(2.0) Loss', 51.805187)], overall loss: -49.096370697021484
Iteration: 352, named_losses: [('ActivationMax Loss', -104.19855),
 ('L-6.0 Norm Loss', 0.062583208),
 ('TV(2.0) Loss', 53.837906)], overall loss: -50.298057556152344
Iteration: 353, named_losses: [('ActivationMax Loss', -102.54213),
 ('L-6.0 Norm Loss', 0.062583096),
 ('TV(2.0) Loss', 52.72509)], overall loss: -49.75445556640625
Iteration: 354, named_losses: [('ActivationMax Loss', -104.47378),
 ('L-6.0 Norm Loss', 0.062587343),
 ('TV(2.0) Loss', 52.638504)], overall loss: -51.77268981933594
Iteration: 355, named_losses: [('ActivationMax Loss', -103.96056),
 ('L-6.0 Norm Loss', 0.062588513),
 ('TV(2.0) Loss', 52.503098)], overall loss: -51.394866943359375
Iteration: 356, named_losses: [('ActivationMax Loss', -105.83009),
 ('L-6.0 Norm Loss', 0.062593691),
 ('TV(2.0) Loss', 53.032692)], overall loss: -52.73480224609375
Iteration: 357, named_losses: [('ActivationMax Loss', -101.99697),
 ('L-6.0 Norm Loss', 0.062594183),
 ('TV(2.0) Loss', 52.436043)], overall loss: -49.49833679199219
Iteration: 358, named_losses: [('ActivationMax Loss', -105.07157),
 ('L-6.0 Norm Loss', 0.062598877),
 ('TV(2.0) Loss', 54.085701)], overall loss: -50.92327117919922
Iteration: 359, named_losses: [('ActivationMax Loss', -104.69636),
 ('L-6.0 Norm Loss', 0.062598661),
 ('TV(2.0) Loss', 52.600704)], overall loss: -52.03305435180664
Iteration: 360, named_losses: [('ActivationMax Loss', -102.88193),
 ('L-6.0 Norm Loss', 0.062602691),
 ('TV(2.0) Loss', 54.546959)], overall loss: -48.272369384765625
Iteration: 361, named_losses: [('ActivationMax Loss', -100.8948),
 ('L-6.0 Norm Loss', 0.062604226),
 ('TV(2.0) Loss', 52.933876)], overall loss: -47.8983154296875
Iteration: 362, named_losses: [('ActivationMax Loss', -106.99043),
 ('L-6.0 Norm Loss', 0.062607288),
 ('TV(2.0) Loss', 54.290199)], overall loss: -52.63761901855469
Iteration: 363, named_losses: [('ActivationMax Loss', -103.61485),
 ('L-6.0 Norm Loss', 0.062607795),
 ('TV(2.0) Loss', 52.990082)], overall loss: -50.562156677246094
Iteration: 364, named_losses: [('ActivationMax Loss', -100.68787),
 ('L-6.0 Norm Loss', 0.06261161),
 ('TV(2.0) Loss', 54.714729)], overall loss: -45.9105224609375
Iteration: 365, named_losses: [('ActivationMax Loss', -106.99897),
 ('L-6.0 Norm Loss', 0.062613629),
 ('TV(2.0) Loss', 52.615471)], overall loss: -54.320884704589844
Iteration: 366, named_losses: [('ActivationMax Loss', -103.92987),
 ('L-6.0 Norm Loss', 0.062616482),
 ('TV(2.0) Loss', 54.538425)], overall loss: -49.32883071899414
Iteration: 367, named_losses: [('ActivationMax Loss', -105.98104),
 ('L-6.0 Norm Loss', 0.062618174),
 ('TV(2.0) Loss', 53.373238)], overall loss: -52.545188903808594
Iteration: 368, named_losses: [('ActivationMax Loss', -103.64502),
 ('L-6.0 Norm Loss', 0.062621377),
 ('TV(2.0) Loss', 54.950352)], overall loss: -48.63204574584961
Iteration: 369, named_losses: [('ActivationMax Loss', -104.32867),
 ('L-6.0 Norm Loss', 0.06262406),
 ('TV(2.0) Loss', 53.403465)], overall loss: -50.862579345703125
Iteration: 370, named_losses: [('ActivationMax Loss', -105.7364),
 ('L-6.0 Norm Loss', 0.062625483),
 ('TV(2.0) Loss', 55.111313)], overall loss: -50.562469482421875
Iteration: 371, named_losses: [('ActivationMax Loss', -105.63428),
 ('L-6.0 Norm Loss', 0.062628642),
 ('TV(2.0) Loss', 54.344261)], overall loss: -51.227386474609375
Iteration: 372, named_losses: [('ActivationMax Loss', -106.10551),
 ('L-6.0 Norm Loss', 0.062629871),
 ('TV(2.0) Loss', 54.804443)], overall loss: -51.238433837890625
Iteration: 373, named_losses: [('ActivationMax Loss', -101.792),
 ('L-6.0 Norm Loss', 0.062634706),
 ('TV(2.0) Loss', 54.769073)], overall loss: -46.960289001464844
Iteration: 374, named_losses: [('ActivationMax Loss', -108.72593),
 ('L-6.0 Norm Loss', 0.06263452),
 ('TV(2.0) Loss', 54.41177)], overall loss: -54.251522064208984
Iteration: 375, named_losses: [('ActivationMax Loss', -102.58292),
 ('L-6.0 Norm Loss', 0.062639162),
 ('TV(2.0) Loss', 54.508747)], overall loss: -48.011539459228516
Iteration: 376, named_losses: [('ActivationMax Loss', -106.88708),
 ('L-6.0 Norm Loss', 0.062639117),
 ('TV(2.0) Loss', 54.107056)], overall loss: -52.71739196777344
Iteration: 377, named_losses: [('ActivationMax Loss', -101.7613),
 ('L-6.0 Norm Loss', 0.062643617),
 ('TV(2.0) Loss', 55.341232)], overall loss: -46.357421875
Iteration: 378, named_losses: [('ActivationMax Loss', -108.98538),
 ('L-6.0 Norm Loss', 0.062643617),
 ('TV(2.0) Loss', 54.409901)], overall loss: -54.51283645629883
Iteration: 379, named_losses: [('ActivationMax Loss', -97.846069),
 ('L-6.0 Norm Loss', 0.062646568),
 ('TV(2.0) Loss', 54.34832)], overall loss: -43.43510437011719
Iteration: 380, named_losses: [('ActivationMax Loss', -108.46047),
 ('L-6.0 Norm Loss', 0.062648959),
 ('TV(2.0) Loss', 53.596687)], overall loss: -54.80113220214844
Iteration: 381, named_losses: [('ActivationMax Loss', -101.88675),
 ('L-6.0 Norm Loss', 0.062652618),
 ('TV(2.0) Loss', 54.373928)], overall loss: -47.45016860961914
Iteration: 382, named_losses: [('ActivationMax Loss', -108.50084),
 ('L-6.0 Norm Loss', 0.062655278),
 ('TV(2.0) Loss', 53.952881)], overall loss: -54.48530578613281
Iteration: 383, named_losses: [('ActivationMax Loss', -102.83924),
 ('L-6.0 Norm Loss', 0.06265676),
 ('TV(2.0) Loss', 54.585762)], overall loss: -48.190818786621094
Iteration: 384, named_losses: [('ActivationMax Loss', -105.97607),
 ('L-6.0 Norm Loss', 0.062660784),
 ('TV(2.0) Loss', 53.984589)], overall loss: -51.92882537841797
Iteration: 385, named_losses: [('ActivationMax Loss', -105.50748),
 ('L-6.0 Norm Loss', 0.062662669),
 ('TV(2.0) Loss', 53.861729)], overall loss: -51.58308792114258
Iteration: 386, named_losses: [('ActivationMax Loss', -105.86003),
 ('L-6.0 Norm Loss', 0.062666699),
 ('TV(2.0) Loss', 54.664692)], overall loss: -51.13267135620117
Iteration: 387, named_losses: [('ActivationMax Loss', -106.63631),
 ('L-6.0 Norm Loss', 0.062667415),
 ('TV(2.0) Loss', 55.014973)], overall loss: -51.55866622924805
Iteration: 388, named_losses: [('ActivationMax Loss', -106.81866),
 ('L-6.0 Norm Loss', 0.062668435),
 ('TV(2.0) Loss', 54.628883)], overall loss: -52.127113342285156
Iteration: 389, named_losses: [('ActivationMax Loss', -105.30239),
 ('L-6.0 Norm Loss', 0.062670067),
 ('TV(2.0) Loss', 55.099808)], overall loss: -50.139915466308594
Iteration: 390, named_losses: [('ActivationMax Loss', -106.9127),
 ('L-6.0 Norm Loss', 0.062673233),
 ('TV(2.0) Loss', 54.387032)], overall loss: -52.46299743652344
Iteration: 391, named_losses: [('ActivationMax Loss', -104.8559),
 ('L-6.0 Norm Loss', 0.06267608),
 ('TV(2.0) Loss', 56.005135)], overall loss: -48.78809356689453
Iteration: 392, named_losses: [('ActivationMax Loss', -106.69425),
 ('L-6.0 Norm Loss', 0.062678099),
 ('TV(2.0) Loss', 54.738026)], overall loss: -51.893550872802734
Iteration: 393, named_losses: [('ActivationMax Loss', -104.99951),
 ('L-6.0 Norm Loss', 0.062681362),
 ('TV(2.0) Loss', 55.773602)], overall loss: -49.16322708129883
Iteration: 394, named_losses: [('ActivationMax Loss', -106.2902),
 ('L-6.0 Norm Loss', 0.06268435),
 ('TV(2.0) Loss', 55.120068)], overall loss: -51.10744857788086
Iteration: 395, named_losses: [('ActivationMax Loss', -107.89492),
 ('L-6.0 Norm Loss', 0.062686883),
 ('TV(2.0) Loss', 55.866585)], overall loss: -51.96565246582031
Iteration: 396, named_losses: [('ActivationMax Loss', -106.80767),
 ('L-6.0 Norm Loss', 0.062688902),
 ('TV(2.0) Loss', 55.007759)], overall loss: -51.737220764160156
Iteration: 397, named_losses: [('ActivationMax Loss', -107.77193),
 ('L-6.0 Norm Loss', 0.062692299),
 ('TV(2.0) Loss', 56.297436)], overall loss: -51.411800384521484
Iteration: 398, named_losses: [('ActivationMax Loss', -105.28301),
 ('L-6.0 Norm Loss', 0.062693223),
 ('TV(2.0) Loss', 55.624161)], overall loss: -49.596160888671875
Iteration: 399, named_losses: [('ActivationMax Loss', -104.94192),
 ('L-6.0 Norm Loss', 0.062694907),
 ('TV(2.0) Loss', 54.875618)], overall loss: -50.00360107421875
Iteration: 400, named_losses: [('ActivationMax Loss', -105.68507),
 ('L-6.0 Norm Loss', 0.062698044),
 ('TV(2.0) Loss', 54.563961)], overall loss: -51.058406829833984
Iteration: 401, named_losses: [('ActivationMax Loss', -109.3108),
 ('L-6.0 Norm Loss', 0.062699236),
 ('TV(2.0) Loss', 55.86681)], overall loss: -53.381290435791016
Iteration: 402, named_losses: [('ActivationMax Loss', -105.72741),
 ('L-6.0 Norm Loss', 0.062703282),
 ('TV(2.0) Loss', 55.159447)], overall loss: -50.50525665283203
Iteration: 403, named_losses: [('ActivationMax Loss', -107.89124),
 ('L-6.0 Norm Loss', 0.062703572),
 ('TV(2.0) Loss', 56.728516)], overall loss: -51.10002136230469
Iteration: 404, named_losses: [('ActivationMax Loss', -106.97041),
 ('L-6.0 Norm Loss', 0.062708952),
 ('TV(2.0) Loss', 55.729668)], overall loss: -51.17803955078125
Iteration: 405, named_losses: [('ActivationMax Loss', -109.42888),
 ('L-6.0 Norm Loss', 0.062708601),
 ('TV(2.0) Loss', 56.222954)], overall loss: -53.143218994140625
Iteration: 406, named_losses: [('ActivationMax Loss', -105.79642),
 ('L-6.0 Norm Loss', 0.062714331),
 ('TV(2.0) Loss', 56.229706)], overall loss: -49.503997802734375
Iteration: 407, named_losses: [('ActivationMax Loss', -109.31998),
 ('L-6.0 Norm Loss', 0.062714919),
 ('TV(2.0) Loss', 55.28862)], overall loss: -53.968650817871094
Iteration: 408, named_losses: [('ActivationMax Loss', -105.02734),
 ('L-6.0 Norm Loss', 0.062719792),
 ('TV(2.0) Loss', 55.842659)], overall loss: -49.12195587158203
Iteration: 409, named_losses: [('ActivationMax Loss', -109.5065),
 ('L-6.0 Norm Loss', 0.062720194),
 ('TV(2.0) Loss', 54.90604)], overall loss: -54.53773880004883
Iteration: 410, named_losses: [('ActivationMax Loss', -105.18683),
 ('L-6.0 Norm Loss', 0.06272237),
 ('TV(2.0) Loss', 55.428093)], overall loss: -49.696014404296875
Iteration: 411, named_losses: [('ActivationMax Loss', -110.12842),
 ('L-6.0 Norm Loss', 0.062723689),
 ('TV(2.0) Loss', 55.221863)], overall loss: -54.843833923339844
Iteration: 412, named_losses: [('ActivationMax Loss', -101.24929),
 ('L-6.0 Norm Loss', 0.062727034),
 ('TV(2.0) Loss', 55.580421)], overall loss: -45.60614013671875
Iteration: 413, named_losses: [('ActivationMax Loss', -111.1939),
 ('L-6.0 Norm Loss', 0.062727764),
 ('TV(2.0) Loss', 55.058533)], overall loss: -56.07263946533203
Iteration: 414, named_losses: [('ActivationMax Loss', -105.27531),
 ('L-6.0 Norm Loss', 0.062730506),
 ('TV(2.0) Loss', 54.636555)], overall loss: -50.57602310180664
Iteration: 415, named_losses: [('ActivationMax Loss', -109.4584),
 ('L-6.0 Norm Loss', 0.062733352),
 ('TV(2.0) Loss', 55.524475)], overall loss: -53.871192932128906
Iteration: 416, named_losses: [('ActivationMax Loss', -106.14929),
 ('L-6.0 Norm Loss', 0.062733874),
 ('TV(2.0) Loss', 55.191273)], overall loss: -50.89528274536133
Iteration: 417, named_losses: [('ActivationMax Loss', -109.47203),
 ('L-6.0 Norm Loss', 0.062737733),
 ('TV(2.0) Loss', 55.093006)], overall loss: -54.316287994384766
Iteration: 418, named_losses: [('ActivationMax Loss', -103.18411),
 ('L-6.0 Norm Loss', 0.062739499),
 ('TV(2.0) Loss', 55.397232)], overall loss: -47.724143981933594
Iteration: 419, named_losses: [('ActivationMax Loss', -110.15713),
 ('L-6.0 Norm Loss', 0.062743336),
 ('TV(2.0) Loss', 55.252705)], overall loss: -54.841678619384766
Iteration: 420, named_losses: [('ActivationMax Loss', -105.91467),
 ('L-6.0 Norm Loss', 0.062745221),
 ('TV(2.0) Loss', 55.511185)], overall loss: -50.34074401855469
Iteration: 421, named_losses: [('ActivationMax Loss', -107.9613),
 ('L-6.0 Norm Loss', 0.062748559),
 ('TV(2.0) Loss', 55.366379)], overall loss: -52.53216552734375
Iteration: 422, named_losses: [('ActivationMax Loss', -108.59941),
 ('L-6.0 Norm Loss', 0.062748775),
 ('TV(2.0) Loss', 55.860031)], overall loss: -52.67662811279297
Iteration: 423, named_losses: [('ActivationMax Loss', -109.84103),
 ('L-6.0 Norm Loss', 0.062752783),
 ('TV(2.0) Loss', 55.039173)], overall loss: -54.73910140991211
Iteration: 424, named_losses: [('ActivationMax Loss', -107.06067),
 ('L-6.0 Norm Loss', 0.062753826),
 ('TV(2.0) Loss', 55.795361)], overall loss: -51.20255661010742
Iteration: 425, named_losses: [('ActivationMax Loss', -109.01567),
 ('L-6.0 Norm Loss', 0.062758133),
 ('TV(2.0) Loss', 56.116005)], overall loss: -52.83690643310547
Iteration: 426, named_losses: [('ActivationMax Loss', -106.97392),
 ('L-6.0 Norm Loss', 0.062758781),
 ('TV(2.0) Loss', 55.328773)], overall loss: -51.58238983154297
Iteration: 427, named_losses: [('ActivationMax Loss', -108.04253),
 ('L-6.0 Norm Loss', 0.062763348),
 ('TV(2.0) Loss', 56.711407)], overall loss: -51.26836013793945
Iteration: 428, named_losses: [('ActivationMax Loss', -107.65493),
 ('L-6.0 Norm Loss', 0.062763982),
 ('TV(2.0) Loss', 55.941433)], overall loss: -51.65073013305664
Iteration: 429, named_losses: [('ActivationMax Loss', -107.2178),
 ('L-6.0 Norm Loss', 0.062767841),
 ('TV(2.0) Loss', 55.630688)], overall loss: -51.52434158325195
Iteration: 430, named_losses: [('ActivationMax Loss', -108.34653),
 ('L-6.0 Norm Loss', 0.062769979),
 ('TV(2.0) Loss', 56.127033)], overall loss: -52.156734466552734
Iteration: 431, named_losses: [('ActivationMax Loss', -107.93786),
 ('L-6.0 Norm Loss', 0.062774226),
 ('TV(2.0) Loss', 57.054199)], overall loss: -50.820884704589844
Iteration: 432, named_losses: [('ActivationMax Loss', -108.71819),
 ('L-6.0 Norm Loss', 0.062775843),
 ('TV(2.0) Loss', 56.083588)], overall loss: -52.57182312011719
Iteration: 433, named_losses: [('ActivationMax Loss', -109.78461),
 ('L-6.0 Norm Loss', 0.062777407),
 ('TV(2.0) Loss', 56.373402)], overall loss: -53.34843063354492
Iteration: 434, named_losses: [('ActivationMax Loss', -107.23654),
 ('L-6.0 Norm Loss', 0.062781304),
 ('TV(2.0) Loss', 57.176895)], overall loss: -49.996864318847656
Iteration: 435, named_losses: [('ActivationMax Loss', -109.90635),
 ('L-6.0 Norm Loss', 0.062783584),
 ('TV(2.0) Loss', 56.665375)], overall loss: -53.178192138671875
Iteration: 436, named_losses: [('ActivationMax Loss', -107.23616),
 ('L-6.0 Norm Loss', 0.062786773),
 ('TV(2.0) Loss', 57.761532)], overall loss: -49.41183853149414
Iteration: 437, named_losses: [('ActivationMax Loss', -111.56915),
 ('L-6.0 Norm Loss', 0.062788755),
 ('TV(2.0) Loss', 56.053192)], overall loss: -55.45317077636719
Iteration: 438, named_losses: [('ActivationMax Loss', -108.50613),
 ('L-6.0 Norm Loss', 0.062791757),
 ('TV(2.0) Loss', 58.244114)], overall loss: -50.1992301940918
Iteration: 439, named_losses: [('ActivationMax Loss', -110.02199),
 ('L-6.0 Norm Loss', 0.062793188),
 ('TV(2.0) Loss', 56.150307)], overall loss: -53.80889129638672
Iteration: 440, named_losses: [('ActivationMax Loss', -109.37215),
 ('L-6.0 Norm Loss', 0.062797546),
 ('TV(2.0) Loss', 57.385628)], overall loss: -51.923728942871094
Iteration: 441, named_losses: [('ActivationMax Loss', -109.52374),
 ('L-6.0 Norm Loss', 0.062798381),
 ('TV(2.0) Loss', 55.699429)], overall loss: -53.76150894165039
Iteration: 442, named_losses: [('ActivationMax Loss', -107.9464),
 ('L-6.0 Norm Loss', 0.062800877),
 ('TV(2.0) Loss', 57.215942)], overall loss: -50.66766357421875
Iteration: 443, named_losses: [('ActivationMax Loss', -109.6866),
 ('L-6.0 Norm Loss', 0.062802352),
 ('TV(2.0) Loss', 55.145344)], overall loss: -54.478450775146484
Iteration: 444, named_losses: [('ActivationMax Loss', -111.0287),
 ('L-6.0 Norm Loss', 0.062804796),
 ('TV(2.0) Loss', 56.066555)], overall loss: -54.89934158325195
Iteration: 445, named_losses: [('ActivationMax Loss', -107.90283),
 ('L-6.0 Norm Loss', 0.062809594),
 ('TV(2.0) Loss', 55.715912)], overall loss: -52.124107360839844
Iteration: 446, named_losses: [('ActivationMax Loss', -109.49814),
 ('L-6.0 Norm Loss', 0.062812239),
 ('TV(2.0) Loss', 57.183552)], overall loss: -52.251773834228516
Iteration: 447, named_losses: [('ActivationMax Loss', -110.68227),
 ('L-6.0 Norm Loss', 0.062813506),
 ('TV(2.0) Loss', 54.801449)], overall loss: -55.8180046081543
Iteration: 448, named_losses: [('ActivationMax Loss', -108.94007),
 ('L-6.0 Norm Loss', 0.062816523),
 ('TV(2.0) Loss', 56.542183)], overall loss: -52.33507537841797
Iteration: 449, named_losses: [('ActivationMax Loss', -111.17997),
 ('L-6.0 Norm Loss', 0.062815465),
 ('TV(2.0) Loss', 55.422665)], overall loss: -55.69449234008789
Iteration: 450, named_losses: [('ActivationMax Loss', -107.12965),
 ('L-6.0 Norm Loss', 0.062819012),
 ('TV(2.0) Loss', 57.527241)], overall loss: -49.53958511352539
Iteration: 451, named_losses: [('ActivationMax Loss', -114.20942),
 ('L-6.0 Norm Loss', 0.062821075),
 ('TV(2.0) Loss', 56.202126)], overall loss: -57.94447326660156
Iteration: 452, named_losses: [('ActivationMax Loss', -105.47456),
 ('L-6.0 Norm Loss', 0.062825598),
 ('TV(2.0) Loss', 57.156342)], overall loss: -48.255393981933594
Iteration: 453, named_losses: [('ActivationMax Loss', -111.92435),
 ('L-6.0 Norm Loss', 0.062826641),
 ('TV(2.0) Loss', 56.216873)], overall loss: -55.64464569091797
Iteration: 454, named_losses: [('ActivationMax Loss', -104.69746),
 ('L-6.0 Norm Loss', 0.062828578),
 ('TV(2.0) Loss', 55.559486)], overall loss: -49.07514190673828
Iteration: 455, named_losses: [('ActivationMax Loss', -111.02028),
 ('L-6.0 Norm Loss', 0.062832452),
 ('TV(2.0) Loss', 56.535641)], overall loss: -54.42180252075195
Iteration: 456, named_losses: [('ActivationMax Loss', -104.58205),
 ('L-6.0 Norm Loss', 0.062834106),
 ('TV(2.0) Loss', 56.202034)], overall loss: -48.317176818847656
Iteration: 457, named_losses: [('ActivationMax Loss', -113.55389),
 ('L-6.0 Norm Loss', 0.062836953),
 ('TV(2.0) Loss', 56.167721)], overall loss: -57.32333755493164
Iteration: 458, named_losses: [('ActivationMax Loss', -106.56391),
 ('L-6.0 Norm Loss', 0.06284035),
 ('TV(2.0) Loss', 56.912422)], overall loss: -49.588645935058594
Iteration: 459, named_losses: [('ActivationMax Loss', -113.20921),
 ('L-6.0 Norm Loss', 0.062841609),
 ('TV(2.0) Loss', 57.164902)], overall loss: -55.981468200683594
Iteration: 460, named_losses: [('ActivationMax Loss', -105.40962),
 ('L-6.0 Norm Loss', 0.062845714),
 ('TV(2.0) Loss', 56.407288)], overall loss: -48.939491271972656
Iteration: 461, named_losses: [('ActivationMax Loss', -112.20129),
 ('L-6.0 Norm Loss', 0.062847808),
 ('TV(2.0) Loss', 56.351261)], overall loss: -55.787174224853516
Iteration: 462, named_losses: [('ActivationMax Loss', -107.91257),
 ('L-6.0 Norm Loss', 0.062850855),
 ('TV(2.0) Loss', 56.704182)], overall loss: -51.14553451538086
Iteration: 463, named_losses: [('ActivationMax Loss', -115.15175),
 ('L-6.0 Norm Loss', 0.062852472),
 ('TV(2.0) Loss', 56.499252)], overall loss: -58.58964538574219
Iteration: 464, named_losses: [('ActivationMax Loss', -108.10796),
 ('L-6.0 Norm Loss', 0.062855773),
 ('TV(2.0) Loss', 58.021297)], overall loss: -50.023807525634766
Iteration: 465, named_losses: [('ActivationMax Loss', -115.26811),
 ('L-6.0 Norm Loss', 0.062857032),
 ('TV(2.0) Loss', 57.482407)], overall loss: -57.72284698486328
Iteration: 466, named_losses: [('ActivationMax Loss', -106.6949),
 ('L-6.0 Norm Loss', 0.062858894),
 ('TV(2.0) Loss', 58.315922)], overall loss: -48.31612014770508
Iteration: 467, named_losses: [('ActivationMax Loss', -116.90611),
 ('L-6.0 Norm Loss', 0.062860802),
 ('TV(2.0) Loss', 56.679832)], overall loss: -60.163421630859375
Iteration: 468, named_losses: [('ActivationMax Loss', -107.25492),
 ('L-6.0 Norm Loss', 0.06286411),
 ('TV(2.0) Loss', 59.087669)], overall loss: -48.10438537597656
Iteration: 469, named_losses: [('ActivationMax Loss', -114.07761),
 ('L-6.0 Norm Loss', 0.06286519),
 ('TV(2.0) Loss', 57.394619)], overall loss: -56.6201286315918
Iteration: 470, named_losses: [('ActivationMax Loss', -108.92873),
 ('L-6.0 Norm Loss', 0.06286791),
 ('TV(2.0) Loss', 57.561081)], overall loss: -51.304779052734375
Iteration: 471, named_losses: [('ActivationMax Loss', -112.0202),
 ('L-6.0 Norm Loss', 0.062869892),
 ('TV(2.0) Loss', 56.625671)], overall loss: -55.3316650390625
Iteration: 472, named_losses: [('ActivationMax Loss', -112.57296),
 ('L-6.0 Norm Loss', 0.062872157),
 ('TV(2.0) Loss', 56.426163)], overall loss: -56.08392333984375
Iteration: 473, named_losses: [('ActivationMax Loss', -112.61472),
 ('L-6.0 Norm Loss', 0.062876411),
 ('TV(2.0) Loss', 57.7103)], overall loss: -54.841548919677734
Iteration: 474, named_losses: [('ActivationMax Loss', -111.89625),
 ('L-6.0 Norm Loss', 0.062876835),
 ('TV(2.0) Loss', 57.044167)], overall loss: -54.789207458496094
Iteration: 475, named_losses: [('ActivationMax Loss', -111.97568),
 ('L-6.0 Norm Loss', 0.062881991),
 ('TV(2.0) Loss', 57.570309)], overall loss: -54.34248733520508
Iteration: 476, named_losses: [('ActivationMax Loss', -110.69009),
 ('L-6.0 Norm Loss', 0.062881552),
 ('TV(2.0) Loss', 56.895927)], overall loss: -53.73127746582031
Iteration: 477, named_losses: [('ActivationMax Loss', -113.54527),
 ('L-6.0 Norm Loss', 0.062884994),
 ('TV(2.0) Loss', 56.79583)], overall loss: -56.686553955078125
Iteration: 478, named_losses: [('ActivationMax Loss', -110.84167),
 ('L-6.0 Norm Loss', 0.062888138),
 ('TV(2.0) Loss', 57.118713)], overall loss: -53.660072326660156
Iteration: 479, named_losses: [('ActivationMax Loss', -109.92656),
 ('L-6.0 Norm Loss', 0.062890992),
 ('TV(2.0) Loss', 56.615776)], overall loss: -53.247894287109375
Iteration: 480, named_losses: [('ActivationMax Loss', -108.08773),
 ('L-6.0 Norm Loss', 0.062892482),
 ('TV(2.0) Loss', 57.017551)], overall loss: -51.00728988647461
Iteration: 481, named_losses: [('ActivationMax Loss', -111.60623),
 ('L-6.0 Norm Loss', 0.062895432),
 ('TV(2.0) Loss', 56.10601)], overall loss: -55.43732452392578
Iteration: 482, named_losses: [('ActivationMax Loss', -109.29018),
 ('L-6.0 Norm Loss', 0.062895611),
 ('TV(2.0) Loss', 56.655937)], overall loss: -52.57134246826172
Iteration: 483, named_losses: [('ActivationMax Loss', -109.88663),
 ('L-6.0 Norm Loss', 0.062899984),
 ('TV(2.0) Loss', 55.779873)], overall loss: -54.04385757446289
Iteration: 484, named_losses: [('ActivationMax Loss', -112.39915),
 ('L-6.0 Norm Loss', 0.062900379),
 ('TV(2.0) Loss', 56.560589)], overall loss: -55.77566909790039
Iteration: 485, named_losses: [('ActivationMax Loss', -109.16783),
 ('L-6.0 Norm Loss', 0.062905088),
 ('TV(2.0) Loss', 56.444405)], overall loss: -52.6605224609375
Iteration: 486, named_losses: [('ActivationMax Loss', -114.19894),
 ('L-6.0 Norm Loss', 0.06290742),
 ('TV(2.0) Loss', 57.282051)], overall loss: -56.85398864746094
Iteration: 487, named_losses: [('ActivationMax Loss', -107.10372),
 ('L-6.0 Norm Loss', 0.062910274),
 ('TV(2.0) Loss', 57.269386)], overall loss: -49.77142333984375
Iteration: 488, named_losses: [('ActivationMax Loss', -113.4107),
 ('L-6.0 Norm Loss', 0.062912695),
 ('TV(2.0) Loss', 57.008839)], overall loss: -56.33894729614258
Iteration: 489, named_losses: [('ActivationMax Loss', -110.55497),
 ('L-6.0 Norm Loss', 0.062915199),
 ('TV(2.0) Loss', 57.273338)], overall loss: -53.218719482421875
Iteration: 490, named_losses: [('ActivationMax Loss', -110.80879),
 ('L-6.0 Norm Loss', 0.06291806),
 ('TV(2.0) Loss', 57.256161)], overall loss: -53.48971176147461
Iteration: 491, named_losses: [('ActivationMax Loss', -112.70348),
 ('L-6.0 Norm Loss', 0.062921286),
 ('TV(2.0) Loss', 57.676243)], overall loss: -54.96432113647461
Iteration: 492, named_losses: [('ActivationMax Loss', -114.01794),
 ('L-6.0 Norm Loss', 0.062924087),
 ('TV(2.0) Loss', 57.559021)], overall loss: -56.39598846435547
Iteration: 493, named_losses: [('ActivationMax Loss', -112.08775),
 ('L-6.0 Norm Loss', 0.062927067),
 ('TV(2.0) Loss', 57.861259)], overall loss: -54.16356658935547
Iteration: 494, named_losses: [('ActivationMax Loss', -114.05003),
 ('L-6.0 Norm Loss', 0.062929094),
 ('TV(2.0) Loss', 58.196888)], overall loss: -55.790218353271484
Iteration: 495, named_losses: [('ActivationMax Loss', -108.59718),
 ('L-6.0 Norm Loss', 0.062931634),
 ('TV(2.0) Loss', 58.215256)], overall loss: -50.31898498535156
Iteration: 496, named_losses: [('ActivationMax Loss', -113.42902),
 ('L-6.0 Norm Loss', 0.0629334),
 ('TV(2.0) Loss', 57.814308)], overall loss: -55.551780700683594
Iteration: 497, named_losses: [('ActivationMax Loss', -110.8927),
 ('L-6.0 Norm Loss', 0.062937751),
 ('TV(2.0) Loss', 57.778252)], overall loss: -53.051513671875
Iteration: 498, named_losses: [('ActivationMax Loss', -113.44555),
 ('L-6.0 Norm Loss', 0.062939815),
 ('TV(2.0) Loss', 57.498432)], overall loss: -55.88417434692383
Iteration: 499, named_losses: [('ActivationMax Loss', -110.71072),
 ('L-6.0 Norm Loss', 0.062944017),
 ('TV(2.0) Loss', 57.633091)], overall loss: -53.01469039916992
Iteration: 500, named_losses: [('ActivationMax Loss', -113.40936),
 ('L-6.0 Norm Loss', 0.062944733),
 ('TV(2.0) Loss', 57.865257)], overall loss: -55.48115539550781

<matplotlib.image.AxesImage at 0x206a906d978>

We can see that the loss appears to be converging. So more iterations definitely seem to give better output. One way to get crisper results is to use Jitter input_modifier. As the name suggests, Jitter moves pixels around in the image. Lets try this out.

from vis.input_modifiers import Jitter

# 20 is the imagenet category for 'ouzel'
# Jitter 16 pixels along all dimensions to during the optimization process.
img = visualize_activation(model, layer_idx, filter_indices=208, max_iter=500, input_modifiers=[Jitter(16)])
plt.imshow(img)

<matplotlib.image.AxesImage at 0x206a924ccf8>

Look at that! Not only has the conv net captured what it means to be an ouzel, but it also seems to encode for different orientations and scales, a further proof of rotational and scale invariance.

Lets try this for a bunch of other random categories. This will take a while. Go grab a nice cup of coffee and prepare to be amused :)

import numpy as np
categories = np.random.permutation(1000)[:15]

vis_images = []
image_modifiers = [Jitter(16)]
for idx in categories:    
    img = visualize_activation(model, layer_idx, filter_indices=idx, max_iter=500, input_modifiers=image_modifiers)
    
    # Reverse lookup index to imagenet label and overlay it on the image.
    img = utils.draw_text(img, utils.lookup_imagenet_labels(idx))
    vis_images.append(img)

# Generate stitched images with 5 cols (so it will have 3 rows).
plt.rcParams['figure.figsize'] = (50, 50)
stitched = utils.stitch_images(vis_images, cols=5)
plt.axis('off')
plt.imshow(stitched)
plt.show()

Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default

Some of them make sense if you stare at ot for a while. There are ways of improving this. We will cover some ideas for this in the next section. You can come back here and try those out as an exercise.

Visualizing Conv filters

In a CNN, each Conv layer has several learned template matching filters that maximize their output when a similar template pattern is found in the input image. First Conv layer is easy to interpret; simply visualize the weights as an image. To see what the Conv layer is doing, a simple option is to apply the filter over raw input pixels. Subsequent Conv filters operate over the outputs of previous Conv filters (which indicate the presence or absence of some templates), making them hard to interpret.

One way of interpreting them is to generate an input image that maximizes the filter output. This allows us to generate an input that activates the filter.

Lets start by visualizing the second conv layer of vggnet (named as 'block1_conv2'). Here is the VGG16 model for reference.

model.summary()

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         (None, 224, 224, 3)       0         
_________________________________________________________________
block1_conv1 (Conv2D)        (None, 224, 224, 64)      1792      
_________________________________________________________________
block1_conv2 (Conv2D)        (None, 224, 224, 64)      36928     
_________________________________________________________________
block1_pool (MaxPooling2D)   (None, 112, 112, 64)      0         
_________________________________________________________________
block2_conv1 (Conv2D)        (None, 112, 112, 128)     73856     
_________________________________________________________________
block2_conv2 (Conv2D)        (None, 112, 112, 128)     147584    
_________________________________________________________________
block2_pool (MaxPooling2D)   (None, 56, 56, 128)       0         
_________________________________________________________________
block3_conv1 (Conv2D)        (None, 56, 56, 256)       295168    
_________________________________________________________________
block3_conv2 (Conv2D)        (None, 56, 56, 256)       590080    
_________________________________________________________________
block3_conv3 (Conv2D)        (None, 56, 56, 256)       590080    
_________________________________________________________________
block3_pool (MaxPooling2D)   (None, 28, 28, 256)       0         
_________________________________________________________________
block4_conv1 (Conv2D)        (None, 28, 28, 512)       1180160   
_________________________________________________________________
block4_conv2 (Conv2D)        (None, 28, 28, 512)       2359808   
_________________________________________________________________
block4_conv3 (Conv2D)        (None, 28, 28, 512)       2359808   
_________________________________________________________________
block4_pool (MaxPooling2D)   (None, 14, 14, 512)       0         
_________________________________________________________________
block5_conv1 (Conv2D)        (None, 14, 14, 512)       2359808   
_________________________________________________________________
block5_conv2 (Conv2D)        (None, 14, 14, 512)       2359808   
_________________________________________________________________
block5_conv3 (Conv2D)        (None, 14, 14, 512)       2359808   
_________________________________________________________________
block5_pool (MaxPooling2D)   (None, 7, 7, 512)         0         
_________________________________________________________________
flatten (Flatten)            (None, 25088)             0         
_________________________________________________________________
fc1 (Dense)                  (None, 4096)              102764544 
_________________________________________________________________
fc2 (Dense)                  (None, 4096)              16781312  
_________________________________________________________________
predictions (Dense)          (None, 1000)              4097000   
=================================================================
Total params: 138,357,544
Trainable params: 138,357,544
Non-trainable params: 0
_________________________________________________________________

from vis.visualization import get_num_filters

# The name of the layer we want to visualize
# You can see this in the model definition.
layer_name = 'block1_conv2'
layer_idx = utils.find_layer_idx(model, layer_name)

# Visualize all filters in this layer.
filters = np.arange(get_num_filters(model.layers[layer_idx]))

# Generate input image for each filter.
vis_images = []
for idx in filters:
    img = visualize_activation(model, layer_idx, filter_indices=idx)
    
    # Utility to overlay text on image.
    img = utils.draw_text(img, 'Filter {}'.format(idx))    
    vis_images.append(img)

# Generate stitched image palette with 8 cols.
stitched = utils.stitch_images(vis_images, cols=8)    
plt.axis('off')
plt.imshow(stitched)
plt.title(layer_name)
plt.show()

Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
IOPub data rate exceeded.
The notebook server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
`--NotebookApp.iopub_data_rate_limit`.

They mostly seem to match for specific color and directional patterns. Lets try a bunch of other layers. We will randomly visualize 10 filters within various layers.

selected_indices = []
for layer_name in ['block2_conv2', 'block3_conv3', 'block4_conv3', 'block5_conv3']:
    layer_idx = utils.find_layer_idx(model, layer_name)

    # Visualize all filters in this layer.
    filters = np.random.permutation(get_num_filters(model.layers[layer_idx]))[:10]
    selected_indices.append(filters)

    # Generate input image for each filter.
    vis_images = []
    for idx in filters:
        img = visualize_activation(model, layer_idx, filter_indices=idx)

        # Utility to overlay text on image.
        img = utils.draw_text(img, 'Filter {}'.format(idx))    
        vis_images.append(img)

    # Generate stitched image palette with 5 cols so we get 2 rows.
    stitched = utils.stitch_images(vis_images, cols=5)    
    plt.figure()
    plt.axis('off')
    plt.imshow(stitched)
    plt.show()

Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default

Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default

Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default

Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default

We can see how filters evolved to look for simple -> complex abstract patterns.

We also notice that some of the filters in block5_conv3 (the last one) failed to converge. This is usually because regularization losses (total variation and LP norm) are overtaking activation maximization loss (set verbose=True to observe). There are a couple of options to make this work better,

• Different regularization weights.
• Increase number of iterations.
• Add Jitter input_modifier.
• Try with 0 regularization weights, generate a converged image and use that as seed_input with regularization enabled.

I will show a subset of these ideas here. Lets start by adidng Jitter and disabling total variation.

layer_idx = utils.find_layer_idx(model, 'block5_conv3')

# We need to select the same random filters in order to compare the results.
filters = selected_indices[-1]
selected_indices.append(filters)

# Generate input image for each filter.
vis_images = []
for idx in filters:
    # We will jitter 5% relative to the image size.
    img = visualize_activation(model, layer_idx, filter_indices=idx, 
                               tv_weight=0.,
                               input_modifiers=[Jitter(0.05)])

    # Utility to overlay text on image.
    img = utils.draw_text(img, 'Filter {}'.format(idx))    
    vis_images.append(img)

# Generate stitched image palette with 5 cols so we get 2 rows.
stitched = utils.stitch_images(vis_images, cols=5)    
plt.figure()
plt.axis('off')
plt.imshow(stitched)
plt.show()

Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default

We can see how previously unconverged filters show something this time. Lets take a specific output from here and use it as a seed_input with total_variation enabled this time.

# Generate input image for each filter.
new_vis_images = []
for i, idx in enumerate(filters):
    # We will seed with optimized image this time.
    img = visualize_activation(model, layer_idx, filter_indices=idx, 
                               seed_input=vis_images[i],
                               input_modifiers=[Jitter(0.05)])

    # Utility to overlay text on image.
    img = utils.draw_text(img, 'Filter {}'.format(idx))    
    new_vis_images.append(img)

# Generate stitched image palette with 5 cols so we get 2 rows.
stitched = utils.stitch_images(new_vis_images, cols=5)    
plt.figure()
plt.axis('off')
plt.imshow(stitched)
plt.show()

Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default
Failed to lookup font 'FreeSans.ttf', falling back to default

And that, folks, is how we roll :) This trick works pretty well to get those stubborn filters to converge.

Other fun stuff

The API to visualize_activation accepts filter_indices. This is generally meant for multi label classifiers, but nothing prevents us from having some fun.

By setting filter_indices, to multiple output categories, we can generate an input that the network thinks is both those categories. Maybe we can generate a cool looking crab fish. I will leave this as an exersice to the reader. You mgith have to experiment with regularization weights a lot.

Ideally, we can use a GAN trained on imagenet and use the discriminator loss as a regularizer. This is easily done using visualize_activations_with_losses API. If you ever do this, please consider submitting a PR :)

Visualizations without swapping softmax

As alluded at the beginning of the tutorial, we want to compare and see what happens if we didnt swap out softmax for linear activation.

Lets try the ouzel visualization again.

layer_idx = utils.find_layer_idx(model, 'predictions')

# Swap linear back with softmax
model.layers[layer_idx].activation = activations.softmax
model = utils.apply_modifications(model)

img = visualize_activation(model, layer_idx, filter_indices=208, input_modifiers=[Jitter(16)])
plt.rcParams['figure.figsize'] = (18, 6)
plt.imshow(img)

C:\Users\admin\Anaconda3\lib\site-packages\keras\models.py:245: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.
  warnings.warn('No training configuration found in save file: '

<matplotlib.image.AxesImage at 0x20712d7fcc0>

It does not work! The reason is that maximizing an output node can be done by minimizing other outputs. Softmax is weird that way. It is the only activation that depends on other node output(s) in the layer.